Sensor Software Driver Programming

The camera sensor driver acquires data from the camera’s sensor over the CSI bus, in the sensor’s native format.

There are two types of camera programming paths. You must choose one, depending on the camera and your application:

• Camera Core Library Interface

• Direct V4L2 Interface

Camera Core Library Interface

The camera core library provides all of the controls and performs all of the processing between the application and kernel-mode V4L2 drivers. Typical use cases for the camera core library interface include:

• Applications that use the Jetson ISP functionality

• Applications that must convert RGB format to YUV format and perform various post-processing tasks for Bayer sensors

The NVIDIA® Jetson™ Linux Driver Package (L4T) architecture framework with the application and kernel mode V4L2 drivers is shown in the following diagram.

A screenshot of a cell phone Description generated with high confidence

Direct V4L2 Interface

In applications that support a direct V4L2 interface, use this interface to communicate with the NVIDIA V4L2 driver without using the camera core library. Use this path for capturing RAW data from a sensor or for validating sensor drivers.

An application uses the kernel mode V4L2 drivers like this:

A screenshot of a cell phone Description generated with very high confidence

To bring up a Bayer pixel format sensor you must:

• Add an appropriate device tree node in the kernel

• Develop a V4L2 sensor driver

You can use either of two versions of the sensor driver for driver development:

• Version 1.0: The version used in past releases. Some version 1.0 features are not available in later releases.

• Version 2.0: A new version that uses the new Jetson V4L2 Camera Framework to modularize code, simplify sensor driver architecture, and encapsulate redundant code. NVIDIA recommends using this version for any new driver development.

The examples provided use Sony IMX185 sensors. Source code for the Sony IMX185 sensor drivers is in these files:

• Version 1.0 driver: imx185_v1.c

• Version 2.0 driver: imx185.c

Camera Modules and the Device Tree

A camera module attached to the target platform can consist of one or more devices. A typical rear camera module includes a complementary metal-oxide semiconductor (CMOS) sensor. A typical front camera module may include a single CMOS sensor.

To add camera modules to a device tree

1. Locate or create a tegra-camera-platform device tree node in the kernel source tree at:

<top>/hardware/nvidia/platform/t19x/common/kernel-dts/t19x-common-modules/tegra194-camera-imx185-a00.dtsi

Where <top> is the directory where you installed the board support package.

2. In a tegra-camera-platform device tree node, create a module table (modules) with one or more modules.

Each module must contain its basic information and the definition of the devices that are inside that module.

This is a typical device tree node for a camera module:

tegra-camera-platform {

compatible = "nvidia, tegra-camera-platform";

modules {

module0 {

badge = "imx185_bottom_liimx185";

position = "bottom";

orientation = "0";

drivernode0 {

pcl_id = "v4l2_sensor";

devname = "imx185 30-001a";

proc-device-tree =

"/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx185_a@1a";

};

Module Properties

The information for moduleX: module (or moduleX: modules) is:

Property	Value
badge	A unique name that identifies this module. The name must consist of three parts separated by underscores: • The module’s camera board ID (camera_board_id). • The position of the module, for example, rear or front. • The module’s part number, which you can find in the module datasheet. If the part number is not available, use a unique identifier. Only the last six characters of the part number are significant. For example, imx185_rear_liimx185 represents a module with a rear-facing IMX185 camera and the part number “llimx185.” If your system has multiple identical modules, each module must have a different position, making the module name unique.
position	Camera position. Values supported depend on the number of cameras in the system: • In a two-camera system: rear and front. • In a three-camera system: bottom, top, and center. • In a six-camera system: bottomleft, bottomright, centerleft, centerright, topleft, and topright.
orientation	Sensor orientation; a numeric index representing the orientation of the sensor. The relation between indices and orientations is arbitrary, but typically in two-camera systems 0 is “rear facing” and 1 is “front facing.”

Individual Imaging Device

An imaging device is a component inside the camera module. It can be:

• A sensor

• A focuser

• A flash

Information must be added to the device tree node to support the device’s operation.

For each device tree node for a device, assign a device tree node that contains:

• The device’s name

• The device’s slave address (available from the device datasheet)

• A compatible property that identifies the node

Note:

Except for devices that refer to other device tree nodes, all value fields in camera-related device tree nodes must use the string data type.

This is an example of a device tree node for the IMX185 V4L2 sensor driver:

imx185_a@1a {

compatible = "nvidia,imx185";

reg = <0x1a>;

physical_w = "15.0";

physical_h = "12.5";

sensor_model ="imx185";

post_crop_frame_drop = "0";

use_decibel_gain = "true";

delayed_gain = "true";

use_sensor_mode_id = "true";

mode0 {

mclk_khz = "37125";

num_lanes = "4";

tegra_sinterface = "serial_a";

phy_mode = "DPHY";

discontinuous_clk = "no";

dpcm_enable = "false";

cil_settletime = "0";

dynamic_pixel_bit_depth = "12";

csi_pixel_bit_depth = "12";

mode_type = "bayer";

pixel_phase = "rggb";

active_w = "1920";

active_h = "1080";

readout_orientation = "0";

line_length = "2200";

inherent_gain = "1";

mclk_multiplier = "2";

pix_clk_hz = "74250000";

gain_factor = "10";

min_gain_val = "0"; /* 0dB */

max_gain_val = "480"; /* 48dB */

step_gain_val = "3"; /* 0.3 */

default_gain = "0";

min_hdr_ratio = "1";

max_hdr_ratio = "1";

framerate_factor = "1000000";

min_framerate = "1500000";

max_framerate = "30000000"; /* 30 */

min_exp_time = "30"; /* us */

max_exp_time = "660000"; /* us */

step_exp_time = "1";

default_exp_time = "33334"; /* us */

embedded_metadata_height = "1";

};

. . .

ports {

#address-cells = <1>;

#size-cells = <0>;

port@0 {

reg = <0>;

liimx185_imx185_out0: endpoint {

port-index = <0>;

bus-width = <4>;

remote-endpoint = <&liimx185_csi_in0>;

};

Device Properties

The following table describes the device tree node properties for the V4L2 sensor device.

Properties in this table can be specified in different DTSI files. For example, clock, GPIO, and regulator properties can be specified in the platform DTSI file and the rest of the properties can be specified in the sensor DTSI files.

Property	Value
compatible	Device identifier. The Linux kernel uses this keyword to bind the device driver to a specific device.
reg	I2C slave address.
mclk*	Name of the input clock for the device. The default value is extperiph1. The maximum frequency for extperiph1 is 24 MHz. If a frequency greater than 24 MHz is required, use an external clock source. Not needed if the sensor is using the clock source provided by the camera module.
<xxx>-gpio*	General-purpose input/output (GPIO) pins for the device, where <xxx> identifies the GPIO pin for the camera. Default GPIO pins for the camera are: • H3-gpio: Camera0 Reset • H6-gpio: Camera0 Power Down • T6-gpio: Camera1 Reset • T5-gpio: Camera1 Power Down Not needed if the sensor has its own I/O control mechanism and does not require specific I/O controls from Jetson.
<xxx>-supply*	Specifies the regulator for the device, where <xxx> is the regulator name for the sensor driver. The value is a reference to the actual regulator name (defined elsewhere in the device tree files), enclosed in angle brackets. The defined regulators are: • vana-supply: Value must be <&en_vdd_cam_hv_2v8> (analog 2.8v). • vdig-supply: Value must be <&en-vdd-cam_1v2> (digital 1.2v). • vif-supply: Value must be <&en-vdd-cam> (interface 1.8v). • vvcm-supply: Value must be <&en_vdd_vcm_2v8> (analog 2.8v for vcm). Not needed if the sensor is using the power sources provided by the camera module.
<xxx>-reg	A unique identifier for the regulator, where <xxx> is the regulator name for the sensor driver. The value of this field is the regulator name with the suffix -reg. The defined regulators are: • avdd-reg: vana • dvdd-reg: vdig • iovdd-reg: vif • vcmvdd-reg: vvcm
physical_w	Physical width of the sensor in millimeters.
physical_h	Physical height of the sensor in millimeters.
sensor_model*	Type of sensor in this module, e.g. "imx185".
set_mode_delay_ms*	Maximum wait time for the first frame after capture starts, in milliseconds.
post_crop_frame_drop*	Number of frames the driver is to drop after applying sensor crop settings. Some sensors must drop frames to avoid receiving corrupted frames after applying crop settings.
use_decibel_gain*	A Boolean, default value false. Determines whether the driver returns analog gain in decibels (dB), according to the formula: dB = 20 * log(Analog Gain) If false, analog gain received by the driver is returned without change.
delayed_gain*	A Boolean, default value false. If true, the user mode driver delays sending updated gain values by one frame. If false, it sends updated gain values to the driver immediately.
use_sensor_mode_id*	A Boolean, default value false. If true, the user mode driver bypasses the default mode selection logic and uses the TEGRA_CAMERA_CID_SENSOR_MODE_ID control to select a specific sensor mode. If false, it uses default mode selection logic, which selects a mode based on resolution, color format, and frame rate.
* Optional

Property-Value Pairs

This table describes the property-value pairs that apply to the sensor mode for the V4L2 implementation.

All properties are set per-mode, and must be set precisely.

Property	Value
mode<n>	Specifies the sensor mode that this group of properties applies to, where <n> is a 0-based index.
ports	Media controller graph binding information. For more information, see Port Binding.
mclk_khz	Standard MIPI driving clock frequency, in kilohertz.
num_lanes	Number of lane channels the sensor is programmed to output.
tegra_sinterface	Base NVIDIA® Tegra® serial interface to which the lanes are connected, for example, "serial_a".
discontinuous_clk	A Boolean; indicates whether the sensor is programmed to use a discontinuous clock on MIPI lanes.
cil_settletime	THS settle time of the MIPI lane, in nanoseconds. Calculate the range of acceptable values with the appropriate formula. • For NVIDIA® Jetson™ Nano devices and Jetson TX1: Where one lp clock period is 1/(102 MHz). • For NVIDIA® Jetson AGX Xavier™ series and Jetson TX2 series: Where one lp clock period is 1/(204 MHz). UI is the unit interval, equal to the duration of the HS state on the clock lane. If the value is 0, the driver attempts to auto-calibrate according to the mclk_multiplier parameter. See the MIPI Alliance Specification for D-PHY for details.
dpcm_enable	A Boolean; specifies whether to enable DPCM compression for this mode.
active_h	Specifies the height of the pixel active region.
active_w	Specifies the width of the pixel active region.
pixel_t	Deprecated. Use the following properties instead: • mode_type • csi_pixel_bit_depth • pixel_phase
mode_type	Sensor mode type. Possible values are: • yuv • bayer • bayer_wdr_pwl: Wide Dynamic Range mode. Uses the piece-wise linear function to compress multi-exposure fused pixel data. Multi-exposure fusion is also performed on the sensor. For more information, see the IMX185 Sensor Driver documentation.
readout_orientation	Specifies clockwise rotation in degrees of the sensor image presented on the display. Intended for use in an Android environment; has no effect with Jetson Linux Driver Package. This property is used to ensure that the sensor image display is oriented correctly for the end user. For example, suppose the sensor is rotated 90° counterclockwise, so that the top of the image falls on the right side of the sensor’s imaging surface. The image on the display appears to be rotated 90° clockwise, with the top of the sensor’s imaging area at the top of the display, and the top of the image on the right. To correct this, set readout_orientation=270. This rotates the displayed image 270° clockwise (90° counterclockwise), rotating the top of the image from the right side of the display to the top.
csi_pixel_bit_depth	Bit depth of sensor output on the CSI bus. If mode_type is set to bayer_wdr_pwl, this property represents the bit depth of sensor output after piece-wise linear function based compression. For more information, see the IMX185 Sensor Driver documentation.
dynamic_pixel_bit_depth*	True dynamic range of the signal. If mode type is set to bayer_wdr_pwl, specifies the bit depth of the multi-exposure fused output before piece-wise linear function based compression. For other mode types, must be equal to csi_pixel_bit_depth. For more information, see the IMX185 Sensor Driver documentation.
pixel_phase*	Sensor pixel phase. Possible values are:
	• uyvy • vyuy • yuyv	• yvyu • rggb • bggr	• grbg • gbrg
	For more information, consult to the IMX185 Sensor Driver documentation.
line_length	Pixel line width horizontal timing size for the sensor mode. Used to calibrate the features in the camera stack. The value must be greater than or equal to active_w.
mclk_multiplier (deprecated)	MCLK multiplier for timing the capture sequence of the hardware. Calculate this value with the equation: mclk_multiplier = desired ISP clock frequency in Hertz / mclk The value must be greater than pixel_clk_hz divided by mclk to prevent ISP underrun. This switch is no longer needed because the value it specifies is calculated internally.
pix_clk_hz	Sensor pixel clock for calculating exposure, frame rate, etc. Calculate the value based on the input clock (mclk) and PLL settings from sensor mode table. See the sensor data sheet for detailed instructions. For more information, see Sensor Pixel Clock.
serdes_pix_clk_hz	Output clock rate for the serializer/deserializer (GMSL or FPD link), in Hertz. Required only for camera modules that use SerDes. For more information, see SerDes Pixel Clock.
inherent_gain	Gain obtained inherently from the mode, that is, pixel binning. May be set to 1 if you do not know the actual value.
min_gain_val	Minimum gain limit for the mode. The value is rounded down to six decimal places. Usually set to 1.0, representing 1x gain.
max_gain_val	Maximum gain limit for the mode. The value is truncated to six decimal places. If supported by the sensor, it may be increased to include digital gain.
min_exp_time	Specifies the minimum exposure time limit for the mode, in microseconds. The value is rounded up to an integer. Calculate the value according to this equation: min_exp_time = (minimum coarse integration time) * line_ length / pix_clk_hz * 1000000 The minimum coarse integration time is the minimum exposure intervals. Unit in lines.
max_exp_time	Specifies the maximum exposure time limit for the mode, in microseconds. The value is rounded up to an integer. Calculate this value with the equation: max_exp_time = (maximum coarse integration time) * line_length / pix_clk_hz * 1000000 The maximum coarse integration time is the maximum exposure intervals. Unit in lines.
min_hdr_ratio	Minimum high-dynamic-range (HDR) ratio denominator for the mode for HDR sensors. Must be an integer greater than 0. For non-HDR sensors, set to 1.
max_hdr_ratio	Maximum HDR ratio denominator for the mode for HDR sensors. Must be an integer greater than 0. For non-HDR sensors, set to 1.
min_framerate	Minimum frame rate limit for the mode, in frames/second (fps). The value is rounded down to six decimal places. Calculate this value with the equation: min_framerate = pix_clk_hz / (line_length * maximum frame length)
max_framerate	Maximum frame rate limit for the mode in fps. The value is rounded down to six decimal places. Calculate this value with the equation: max_framerate = pix_clk_hz / (line_length * minimum frame length)
embedded_metadata_height	Specifies the number of extra embedded metadata rows for each frame. Set to 0 to disable embedded metadata support.
num_control_point*	Number of control points used in the piece-wise linear function for compressing multi-exposure fused pixel data. Multi-exposure fusion is also performed on the sensor. Valid if mode_type is set to bayer_wdr_pwl. The camera core library interface supports up to 9 control points. For more information, see the IMX185 Sensor Driver documentation.
control_point_x_[<n>]*	X coordinate of control point <n> from the piece-wise linear compression function, where <n> is an index from 0 to 15. (Do not include angle brackets in an actual property name.) Valid if mode_type is set to bayer_wdr_pwl. The value may not exceed 2dynamic_pixel_width_depth.
control_point_y_ [<n>]*	Y coordinate of control point <n> from the piece-wise linear compression function, where <n> is an index from 0 to 15. (Do not include angle brackets in an actual property name.) Valid if mode_type is set to bayer_wdr_pwl. The value may not exceed 2csi_pixel_width_depth.
For WDR DOL (Digital Overlap) sensors
num_of_exposure†	Number of exposure frames in a DOL multi-frame.
num_of_ignored_lines†	Number lines to crop at the top of each exposure frame. Usually includes the OB (Optical Black) rows and ignored area of effective pixel rows, both of which use the same LI (line information) header as the exposure frame on which they reside. The VI (video input) module handles cropping for these lines.
num_of_lines_offset_0†	Number of VBP rows appearing in a single exposure frame of a two-exposure DOL multi-frame. These VBP rows are filtered out as they have LI (line information) indicating that they are blank. Note that the same number of VBP rows appear at the end of a long exposure frame and at the start of a short exposure frame. For DOL mode with two exposure frames, there is only one type of VBP row. With more than two exposure frames, there may be different types of VBP rows. Hence the parameter can be extended to number_of_offset_1, number_of_offset_2,…. The number of VBP row types is usually one less than the number of exposure frames in a DOL frame.
num_of_ignored_pixels†	Size of LI (line information) header for each row, in pixels. The line information helps differentiate VBP (Vertical Blank Period) rows and different exposure rows such as long exposure and short exposure. The VI (video input) module handles cropping for these lines.
num_of_left_margin_pixels†	Size of the left margin, before the active pixel area, in pixels. These lines are required for cropping alignment. The VI (video input) module handles cropping for them.
num_of_right_margin_pixels†	Size of the right margin, after the active pixel area, in pixels. These lines are required for cropping alignment. The VI (video input) module handles cropping for them.
* Alternative to pixel_t when WDR PWL sensor is used. † When WDR DOL sensor is used. See the diagram below.

Property-Value Pairs (Version 2.0 Only)

These properties are supported for version 2.0 sensor drivers only, and should not be used with version 1.0 drivers.

Property	Value
gain_factor	Multiplication factor for gain control to represent float values in fixed point. Choose a gain factor which ensures that the range and delta of gain can be configured as required by the sensor driver.
default_gain	Default gain value for gain control with the specified mode. default_gain is an integer value equal to the actual default gain scaled by gain_factor: default_gain = int(default gain in float * gain_factor)
step_gain_val	Minimum delta value to which gain control can be adjusted.
exposure_factor	Multiplication factor for exposure control to represent floating point values in fixed point. Choose an exposure_factor which ensures that the range and delta of exposure can be configured as required by the sensor driver.
default_exp_time	Default exposure value to be used for exposure control with the specified mode. default_exp_time is an integer value equal to the default exposure time scaled by exposure_factor: default_exp_time = int(default exposure time in float * exposure_factor)
step_exp_time	Minimum delta value to which the exposure control can be adjusted.
framerate_factor	Multiplication factor for frame rate control to represent float values in fixed point. Choose a framerate factor which ensures that the range and delta of frame rate can be configured as required by the sensor driver.
default_framerate	Default framerate value for framerate control with the specified mode. default_framerate is an integer value equal to the actual default framerate scaled by framerate_factor: default_framerate = int(default framerate in float * framerate_factor)
step_framerate	Minimum delta value to which the framerate control can be adjusted.
min_gain_val	Minimum gain limit for the specified mode. The value specified must be multiplied by gain_factor. min_gain_val = (minimum gain value in float) * gain_factor
max_gain_val	Maximum gain limit for the specified mode. The value specified must be multiplied by gain_factor. max_gain_val = (maximum gain value in float) * gain_factor
min_exp_time	Minimum exposure time limit for the mode, in microseconds. The value is rounded up to an integer. This value is calculated from: minimum exposure time in float = (minimum coarse integration time) * line_length / pix_clk_hz * 1000000 The value specified must be multiplied by exposure_factor. min_exp_time = (minimum exposure time in float) * exposure_factor
max_exp_time	Maximum exposure time limit for the mode, in microseconds. The value is rounded up to an integer. The value is calculated from: maximum exposure time in float = (maximum coarse integration time) * line_length / pix_clk_hz * 1000000 The value specified must be multiplied by exposure_factor. max_exp_time = (maximum exposure time in float) * exposure_factor
min_framerate	Minimum frame rate limit for the mode, in frames/second (fps). The value is calculated from: minimum framerate in float = pix_clk_hz / (line_length * maximum frame length) The value specified must be multiplied by framerate_factor. min_framerate = (minimum framerate in float) * framerate_factor
max_framerate	Maximum frame rate limit for the mode in fps. The value is rounded down to six decimal places. The equation to calculate this value is: maximum framerate in float = pix_clk_hz / (line_length * minimum frame length) The value specified must be multiplied by framerate_factor. max_framerate = (maximum framerate in float) * framerate_factor

Example Piece-wise Linear Compression Function

This diagram shows an example of a piece-wise linear compression function.

In this example:

• Input signal has 16-bit depth

• Output signal has 12-bit depth

• csi_pixel_bit_depth = "12"

• dynamic_pixel_bit_depth = "16"

The following table shows the control point properties.

Property	Value
num_control_point	4
control_point_x_0	0
control_point_x_1	2048
control_point_x_2	16384
control_point_x_3	65536
control_point_y_0	0
control_point_y_1	2048
control_point_y_2	2944
control_point_y_3	3712

Example Digital Overlap WDR Exposure Frame (3840×2160)

This diagram shows an example of a two-exposure DOL (Digital Overlap) multi-frame.

The following table shows the values that apply to the diagram under Example Piece-wise Linear Compression Function for a 4K digital overlap WDR frame.

Property	Value
active_w	3856
active_h	4436
num_of_exposure	2
num_of_ignored_lines	14
num_of_lines_offset_0	50
num_of_ignored_pixels	4
num_of_left_margin_pixels	12
num_of_right_margin_pixels	0

Line Information (LI) header pixels help differentiate Vertical Blank Period (VBP) rows and different types of exposure rows. Thus there are three kinds of LI for the diagram above: VBP, long exposure, and short exposure. VPB rows are filtered out by the VI (video input) module based on LI.

The property values are computed as follows:

• active_w: Total width of the pixel-active region. In this case, it is 3840 + 4 (LI) + 12 (left margin) + 0 (right margin) = 3856.

• active_h: Total height of the pixel-active region. In this case, it is 2160 + 8 (OB) + 6 (Ignored Area Effective Pixel + 50 (VBP)) * 2 = 4448.

• num_of_exposure: Number of exposures in each digital overlap capture. For the above example of 4K digital overlap WDR, this is 2 (two exposures).

• num_of_ignored_lines: Number of rows at the top of each exposure frame that are not included in the output frame. This includes OB rows and Ignored Area Effective Pixel rows, both of which use the same LI header as the exposure frame on which they reside. Hence, these rows are read by the VI module and are cropped out. For the example above, this value is 14 rows = 8 OB rows + 6 Ignored Area Effective Pixel rows.

• num_of_lines_offset_0: Number of VBP rows appearing in a single exposure and frame in a two-exposure DOL multi-frame. The same number of VBP rows appear at the end of a long exposure frame and at the start of a short exposure frame. For the example above, this value is 50 (50 rows at the end of a long exposure frame and 50 rows at the start of a short exposure frame).

• num_of_ignored_pixels: Number of LI pixels at the start of each line that distinguish between different exposure rows and VBP rows. For the example above, this value is 4 (four pixels).

• num_of_left_margin_pixels: Number of left margin pixels before the image data on each row. These pixels are required for cropping alignment. For the example above, this value is 12 (12 pixels).

• num_of_right_margin_pixels: Number of right margin pixels after the image data on each row. These are required for cropping alignment. For the example above, this value is 0 (zero pixels).

Port Binding

The following node definitions demonstrate the port binding for VI, NvCSI, and sensor modules.

vi {

num-channels = <1>;

ports {

#address-cells = <1>;

#size-cells = <0>;

port@0 {

reg = <0>;

liimx185_vi_in0: endpoint {

port-index = <2>;

bus-width = <4>;

remote-endpoint = <&liimx185_csi_out0>;

};

nvcsi {

num-channels = <1>;

#address-cells = <1>;

#size-cells = <0>;

channel@0 {

reg = <0>;

ports {

#address-cells = <1>;

#size-cells = <0>;

port@0 {

reg = <0>;

liimx185_csi_in0: endpoint@0 {

port-index = <0>;

bus-width = <4>;

remote-endpoint = <&liimx185_imx185_out0>;

};

port@1 {

reg = <1>;

liimx185_csi_out0: endpoint@1 {

remote-endpoint = <&liimx185_vi_in0>;

};

The following table shows the port binding properties.

Property	Value
port	Specifies the media pad port connection. All imager devices have one media pad for binding the connection with VI.
port-index	Defines the sensor port connection. This field is not required for imager devices, such as a focuser or flash. For more information, see Port Index.
bus-width	Number of CSI lanes connected to sensor; determines the bus width.
remote-endpoint	Label for binding two ports. The binding expects one port to be for the sink and the other one for the source.

To verify the port binding result

• Enter the command:

sudo media-ctl -p -d /dev/media0

The output returned is similar to this:

Media controller API version 0.1.0

Media device information

------------------------

driver tegra-vi4

model NVIDIA Tegra Video Input Device

serial

bus info

hw revision 0x3

driver version 0.0.0

Device topology

- entity 1: 15a00000.nvcsi--1 (2 pads, 2 links)

type V4L2 subdev subtype Unknown flags 0

device node name /dev/v4l-subdev0

pad0: Sink

<- "imx185 30-001a":0 [ENABLED]

pad1: Source

-> "vi-output, imx185 30-001a":0 [ENABLED]

- entity 4: imx185 30-001a (1 pad, 1 link)

type V4L2 subdev subtype Sensor flags 0

pad0: Source

-> "15a00000.nvcsi--1":0 [ENABLED]

- entity 6: vi-output, imx185 30-001a (1 pad, 1 link)

type Node subtype V4L flags 0

pad0: Sink

<- "15A00000.nvcsi--1":1 [ENABLED]

Sensor Pixel Clock

The camera software uses the sensor pixel clock to calculate the exposure and frame rate of the sensor. It must be set correctly to avoid potential issues.

Depending on the information provided by the sensor vendors, there are several ways to obtain the correct sensor pixel clock rate:

• Using PLL multiplier and PLL pre/post dividers:

MCLK Multiplier = PLL Multiplier / PLL Pre-divider / PLL Post-divider

pixel_clk_hz = MCLK * MCLK Multiplier

For example:

● MCLK = 24MHz

● PLL Multiplier = 30

● PLL Pre-divider = 3

● PLL Post-divider = 3

MCLK Multiplier = 30 / 3 / 3 = 6.67
Pixel_clk_hz = 24000000 * 6.67 = 160000000

• Using sensor CSI lane output rate:

pixel_clk_hz = sensor data rate per lane (Mbps) * number of lanes / bits per pixel

For example:

● Sensor data rate = 891 Mbps (per lane)

● Number of lanes = 4

● Bits per pixel = 10

pixel_clk_hz = 891 Mbps * 4 / 10 = 356400000

• Using frame size and frame rate

pixel_clk_hz = sensor output size * frame rate

Note:

The sensor output size is not the final output size. It is the total number of pixels the sensor used to generate the final output size.

For example:

Sensor output size = 2200 * 1125
(actual output size 1920 * 1080)

Frame rate = 30 FPS

pixel_clk_hz = 2200 * 1125 * 30 = 74250000

SerDes Pixel Clock

For sensor modules that use serializer/deserializer chips (GMSL or FPD link), the frames received by the SoC are output from SerDes, not from the sensors. Therefore, the SerDes pixel clock must be specified correctly to configure the SoC-camera interface correctly and avoid buffer overrun issues.

The SerDes pixel clock must be set based on the use case (on the number of cameras connected to SerDes). If you are not sure which use case to adopt, set it to the SerDes maximum output rate.

Port Index

Port index is used to specify the connection between two different modules. For the VI module, the port index represents the input stream. For the CSI module, it represents the actual CSI port to which the camera is connected.

The following diagrams show the port index mapping for different Jetson platforms.

• For NVIDIA® Jetson Nano™ series, Jetson™ TX2 series, and Jetson TX1::

http://sw-mobile-docs/DRAFT/L4T_32/Tegra%20Linux%20Driver%20Package%20Development%20Guide/images/image5_3.png

• NVIDIA® Jetson AGX Xavier™ series:

V4L2 Kernel Driver (Version 1.0)

This topic is based on the Video for Linux 2 (V4L2) driver for the Sony IMX185 sensor (imx185_v1.c) at:

<top>/kernel/nvidia/drivers/media/i2c/imx185_1.c

Examine this source file to obtain a complete understanding of the driver.

For information on the V4L2 implementation, consult:

https://linuxtv.org/downloads/v4l-dvb-apis/

Macro Definitions

Sensor-specific macro values include:

• The minimum and maximum values for each control

• The default value for each control

• The macro values required for sensor timing or general functionality

Sensor-Private Data

The structure contains the private data specific to the sensor:

struct imx185 {

struct camera_common_power_rail power;

int numctrls;

struct v4l2_ctrl_handler ctrl_handler;

struct i2c_client *i2c_client;

struct v4l2_subdev *subdev;

struct media_pad pad;

u32 frame_length;

s32 group_hold_prev;

bool group_hold_en;

s64 last_wdr_et_val;

struct regmap *regmap;

struct camera_common_data *s_data;

struct camera_common_pdata *pdata;

struct v4l2_ctrl *ctrls[];

};

The following table describes the sensor properties.

Property	Value
power	Generic power controls, including regulators, clocks, and GPIOs.
numctrls	Number of V4L2 controls for the sensor.
ctrl_handler	Holds the V4L2 handler for controls, needed for V4L2 control initialization.
i2c_client	Holds a handle to i2c_client, used to access the sensor’s I2C client instance.
subdev	Holds a handle for the V4L2 subdevice; used to run subdev operations (ops).
pad	Holds a media pad used to initialize the media controller for a device to work as a sink pad or source pad.
group_hold_prev	Holds the previous state. Used by the group hold control handler to check for change of state.
group_hold_en	Holds a group hold enable flag directly related to group hold control.
frame_length	Holds the previous value of frame_length.
reg_map	Holds a register map setup for I2C read and write.
s_data	Holds a handle to common data. For more information, see struct camera_common_data in the file camera_common.h.
pdata	Holds a handle to common platform data, populated by reading the device tree.
ctrls	Holds handles to initialized V4L2 controls and dynamic array. Must be the last property specified.

Configuring Regmap

You must update the sensor_regmap_config structure with appropriate values of reg_bits and val_bits to describe the sensor’s I2C interface: The following table shows the members of the structure.

Property	Value
reg_bits	Size of the register address, in bits.
val_bits	Size of the register value, in bits.

A definition of sensor_regmap_config looks like this:

struct regmap_config sensor_regmap_config {

reg_bits = 16;

val_bits = 8;

};

Check the vendor’s register programming table for your sensor to determine the values for the structure’s members.

Configuring Controls

The controls must be linked to their control handlers.

To link the controls to their control handlers

• Define a static v4l2_ctrl_ops structure. Initialize structure members as follows:

● Point g_volatile_ctrl to the sensor’s internal Get Volatile Control handler.

● Point s_ctrl to the sensor’s internal set control handler.

A definition of the v4l2_ctrl_ops structure looks like this:

static const struct v4l2_ctrl_ops imx185_ctrl_ops = {

.g_volatile_ctrl = imx185_g_volatile_ctrl,

.s_ctrl = imx185_s_ctrl,

};

The following example code lists the controls and their initialized values. The initial ctrls_init() call loops through this list to initialize each of the controls. Each control is then accessible through the ctrls handler in the private data. The set of controls defined here for IMX185 are the standard ones used by the user-mode library.

Note:

To implement additional controls you must change the camera core library.

Three types of controls, integer, menu, and string, are defined for the IMX185 sensor, as shown in the following example code.

static struct v4l2_ctrl_config ctrl_config_list[] = {

/* Integer Control: setting integer values such as gain, coarse

time, and frame length. */

{

.ops = &imx185_ctrl_ops, // Pointer to control ops function.

.id = TEGRA_CAMERA_CID_GAIN,// id, defined in camera_common.h.

.name = "Gain", // String name of control.

.type = V4L2_CTRL_TYPE_INTEGER, // Type of control.

.flags = V4L2_CTRL_FLAG_SLIDER, // Control flags.

/* The following values define the scaling factor, and are

the values most likely to change. */

.min = 0 * FIXED POINT SCALING FACTOR, // Lower bound.

.max = 48 * FIXED POINT SCALING FACTOR, // Upper bound.

.def = 0 * FIXED POINT SCALING FACTOR, // Default.

.step = 3 * FIXED POINT SCALING FACTOR, / 10,

// Increment step size.

/* Menu Control: used as on/off switch for group hold and HDR.

* switch_ctrl_qmenu is used to define the states on/off. You do

* not need to change these controls unless you add a completely

* new one.

* enum switch_state {

* SWITCH_OFF,

* SWITCH_ON,

* };

* static const s64 switch_ctrl_qmenu[] = {

* SWITCH OFF, SWITCH ON

* };

{

.ops = &imx185_ctrl_ops,

.id = TEGRA_CAMERA_CID_GROUP_HOLD,

.name = "Group Hold",

.type = V4L2_CTRL_TYPE_INTEGER_MENU,

.min = 0,

.max = ARRAY_SIZE(switch_ctrl_qmenu) - 1,

.menu_skip_mask = 0,

.def = 0,

.qmenu_int = switch_ctrl_qmenu,

/* String Control: used to get static sensor information like

* EEPROM content or Fuse ID.

{

.ops = &imx185_ctrl_ops,

.id = TEGRA_CAMERA_CID_FUSE_ID,

.name = "Fuse ID",

.type = V4L2_CTRL_TYPE_STRING,

.flags = V4L2_CTRL_FLAG_READ_ONLY,

.min = 0,

.max = IMX185_FUSE_ID_STR_SIZE,

.step = 2,

};

Setting Up Control Registers

You can use the helper functions below to update sensor registers based on the control types. Each function takes two input parameters, an array of register address/value pairs and a control value.

When the control handlers call each function, it updates the address/value pairs specified in the array based on the control types. The array of address/values pairs are used to program the sensor using the register write wrapper functions.

imx185_get_frame_length_regs()

imx185_get_coarse_time_regs_shs1()

imx185_get_coarse_time_regs_shs2()

imx185_get_gain_reg()

Read-Write Wrapper in the Register

The following functions are wrappers for reading and writing sensor registers. You can use them to read or write a single register, or to write a table of registers.

For IMX185, the functions use the regmap I2C interface. You must modify them if your sensor uses some other interface (e.g. SPI).

imx185_read_reg()

imx185_write_reg()

imx185_write_table()

Power Functions

The following table describes the functions for power-related controls.

Function	Description
imx185_power_on()	Implements the sensor power-on sequence. Modify this function according to the sensor specification sheets. The function uses regulator_enable() to turn on regulators, and gpio_set_value() to toggle the GPIO pins.
imx185_power_off()	Implements the sensor power-off sequence. Modify this function according to the sensor specification sheets. The function uses regulator_enable() to turn on regulators, and gpio_set_value() to toggle the GPIO pins.
imx185_power_put()	Releases all resources used by the sensor. The function uses regulator_put() to release resources for all regulators.
imx185_power_get()	Acquires resources used by the sensor. The function uses the camera_common_regulator_get() helper function or regulator_get() to acquire all regulators.

The driver must call imx185_power_get()first to acquire all regulators prior to calling imx185_power_on() and imx185_power_off().

Additionally, the driver must call imx185_power_put()to release all regulators when the driver is being released.

Setting Up the V4L2 Subdevice and Camera Common

The imx185_s_stream() function enables and disables sensor streaming. It programs sensor registers using the mode tables in the file imx185_mode_tbls.h file.

Instead of using the default settings in mode tables, you can also override some of the settings in this function by providing updated values. The code below shows how to override gain, exposure and frame rate values immediately after the mode tables are programmed to sensor.

Control[0].id = TEGRA_CAMERA_CID_GAIN;

Control[1].id = TEGRA_CAMERA _CID_FRAME_RATE;

Control[2].id = TEGRA_CAMERA _CID_EXPOSURE;

err = v4l2_g_ext_ctrls(&priv->ctrl_handler, &cotrl);

err |= imx185_set_gain(priv, control[0].value64);

. . .

If the sensor supports a sensor test pattern, you can enable it in this function by selecting the proper test pattern mode tables.

Camera Common is a set of kernel functions that are used by the camera drivers in the NVIDIA kernel and by V4L2 drivers. Camera Common sets up most of the V4L2 framework. You communicate with it through camera_common_sensor_ops and camera_common_data structures, which you must define in your driver.

For details on modifying and adding new modes, see Mode Tables.

The following code snippets set up the V4L2 subdevices and Camera Common for registering the sensor driver with the V4L2 framework.

The s_stream pointer in the following declaration (highlighted below) must point to the sensor’s implementation of the s_stream() function; you can leave the other pointers as they are.

static struct v4l2_subdev_video_ops imx185_subdev_video_ops = {

.s_stream = imx185_s_stream,

...

};

You do not need to modify this declaration:

static struct v4l2_subdev_core_ops imx185_subdev_core_ops = {

...

};

In the following declaration, initialize the core and video members with references to the two corresponding structures (highlighted below).

static struct v4l2_subdev_ops imx185_subdev_ops = {

.core = &imx185_subdev_core_ops,

.video = &imx185_subdev_video_ops,

};

The compatible member of the of_device_id structure (highlighted below) must match what is specified in sensor’s device tree node. The Linux kernel uses this member to identify and bind the driver to the sensor.

static struct of_device_id imx185_of_match[] = {

{ .compatible = "nvidia, imx185", },

{ },

};

Link this declaration’s power_on, power_off, write_reg, and read_reg members to the appropriate functions (highlighted below):

static struct camera_common_sensor_ops imx185_common_ops = {

.power_on = imx185_power_on,

.power_off = imx185_power_off,

.write_reg = imx185_write_reg,

.read_reg = imx185_read_reg,

};

Control Handlers

This topic describes the control handlers that control the top-level structure and tells the function how many controls the handler is expected to handle.

Set Control

The following table describes controls that may be implemented for a sensor driver. Each control needs a dedicated control handler.

The property values that were originally floating point are scaled to integers by their respective scaling factors. (See, for example, the discussion of Fixed Point Format.)

Property	Input
TEGRA_CAMERA_CID_GAIN	Updates the gain settings. The value is the actual gain scaled by gain_factor.
TEGRA_CAMERA_CID_FRAME_LENGTH or V4L2 CID_FRAME_RATE	Updates the frame length register with the frame length in lines, or the frame rate settings with the frame rate in frames/second.
TEGRA_CAMERA_CID_COARSE_TIME or TEGRA_CAMERA_CID_EXPOSURE	Updates the coarse integration time registers with the time in lines, or the exposure settings with the exposure time in seconds. See Exposure Controls.
TEGRA_CAMERA_CID_COARSE_TIME_SHORT* or TEGRA_CAMERA_CID_EXPOSURE_SHORT*	Updates the short coarse integration time registers with the time in lines, or the short exposure settings with the exposure time in seconds. Only available for HDR sensors.
TEGRA_CAMERA_CID_GROUP_HOLD	Updates the sensor group hold (register hold) control for sensors supporting this feature. Create an empty control handler if sensor hardware does not support this feature.
TEGRA_CAMERA_CID_HDR_ENABLE*	A Boolean; indicates whether the sensor is operating in HDR sensor mode (true) or normal sensor mode (false). Only available for HDR sensors.
TEGRA_CAMERA_CID_EEPROM_DATA	Reads the module’s EEPROM data.
TEGRA_CAMERA_CID_OTP_DATA*	Reads the module’s One-Time Programmable (OTP) memory data.
TEGRA_CAMERA_CID_FUSE_ID*	Reads the sensor fuse ID.
TEGRA_CAMERA_CID_SENSOR_MODE_ID*	Selects specific sensor mode. Value is a zero-based index.
TEGRA_CAMERA_CID_VI_BYPASS_MODE†	A Boolean; true bypasses VI settings in the kernel driver. Default is true when using camera core library.
TEGRA_CAMERA_CID_OVERRIDE_ENABLE**	Overrides gain and exposure settings from the default sensor mode table.
* Optional † Provided by NVIDIA V4L2 Media Controller Framework kernel driver.

Gain Control

Gain control applies to the per-frame gain setting. It takes a 32-bit or 64-bit input value from the camera core library.

Property	Value
TEGRA_CAMERA_CID_GAIN	Specifies the gain values from the camera core library. 32-bit mode (deprecated): Uses 24.8 format (24‑bit integer, 8‑bit fraction) by default. To change the format, adjust the min/max gain range in the driver. Uses the minimum gain value to determine how many bits of a fraction to use. 64-bit mode: Uses 42.22 format (42‑bit integer, 22‑bit fraction). (See Fixed Point Format.) For details, see the definition of imx185_set_gain() in the file imx185.c.

Property

Value

TEGRA_CAMERA_CID_GAIN

Specifies the gain values from the camera core library.

32-bit mode (deprecated): Uses 24.8 format (24‑bit integer, 8‑bit fraction) by default. To change the format, adjust the min/max gain range in the driver.

Uses the minimum gain value to determine how many bits of a fraction to use.

64-bit mode: Uses 42.22 format (42‑bit integer, 22‑bit fraction). (See Fixed Point Format.)

For details, see the definition of imx185_set_gain() in the file imx185.c.

Exposure Control

You exercise exposure control with one of two pairs of controls: one pair for 32‑bit input parameters (deprecated), and one for 64‑bit. You may implement and use either pair of controls, but do not implement both.

Property	Value
TEGRA_CAMERA_CID_FRAME_LENGTH + TEGRA_CAMERA_CID_COARSE_TIME (Deprecated)	Specifies the pre-processed 32-bit value. The camera core library converts the frame rate to the sensor frame length and converts exposure to sensor coarse integration time. The driver uses these values to program the frame length and coarse integration time registers.
TEGRA_CAMERA_CID_FRAME_RATE + V4L2 CID_EXPOSURE	Specifies the unprocessed 64-bit value. (See Fixed Point Format.) The camera core library passes the frame rate and exposure values to the driver with no pre-processing. The driver is responsible for converting these values into frame length and coarse integration time settings. See the definitions of the imx185_set_frame_rate() and imx185_set_exposure() functions in the file imx185_1.c.

Fixed Point Format

If the driver implements the TEGRA_CAMERA_CID_FRAME_RATE and TEGRA_CAMERA_CID_EXPOSURE controls, the received control values are in Q42.22 fixed point format. In this format the upper 42 bits are for storing integer values and the lower 22 bits are for fraction values. This is a workaround for the lack of floating point capability in the kernel.

To convert a floating point number to Q42.22 format, multiply it by 1<<22. To convert it back to floating point format, divide it by 1<<22. The manifest constant FIXED_POINT_SCALING_FACTOR represents this factor:

#define FIXED_POINT_SCALING_FACTOR (1ULL << 22)

To minimize loss of precision, perform multiplication first, then division. For example, these statements yield different results:

u64 gain64 = 10485760; // 2.5 * FIXED_POINT_SCALING_FACTOR.

gain8 = (u8) (gain64 / FIXED_POINT_SCALING_FACTOR * 160 / 48);

// Incorrect, gain8 = 6.

gain8 = (u8) (gain64 * 160 / 48 / FIXED_POINT_SCALING_FACTOR);

// Correct, gain8 = 8.

V4L2 Set-Control Operation

The V4L2 set-control function contains a switch statement to redirect set-control calls to the appropriate control handlers.

Note:

Read-only controls, such as TEGRA_CAMERA_CID_FUSE_ID (which reads the camera’s fused module ID) and TEGRA_CAMERA_CID_OTP_DATA (which reads its One-Time Programmable Read-Only Memory), do not have a case statement in the control ID switch statement.

imx185_s_ctrl {

...

switch (ctrl->id) {

case TEGRA_CAMERA_CID_GAIN:

...

case TEGRA_CAMERA_CID_EEPROM_DATA:

...

case TEGRA_CAMERA_CID_HDR_EN:

break;

}

...

(TEGRA_CAMERA_CID_HDR_EN is a pure software control. No control handler writes to the hardware, so simply break out of the switch statement.)

Setter-Control Handlers (Writes)

Setter-control handlers are the control handlers called by s_ctrl(). They perform additional control-handling operations, such as writes to registers. The majority of these controls make calls to the control register setup functions.

imx185_set_group_hold()

imx185_set_gain()

imx185_set_frame_length()

imx185_set_coarse_time()

imx185_set_coarse_time_short()

imx185_write_eeprom()

For more information, see Setting Up Control Registers.

Get-Volatile Control

imx185_g_volatile_ctrl() is a wrapper function that redirects get-control calls to their appropriate volatile control handlers.

Note:

For non-volatile controls, get-control calls simply return the previously written value stored in the control handler.

Boot-Time Initialization

These sections describe functions for boot-time initialization.

Control Initialization

The imx185_ctrls_init() function iterates through ctrl_config_list[] and registers each control as a new custom control with the V4L2 framework. imx185_s_ctrl() is also called for each control to initialize its default value as defined in ctrl_config_list.

For more information, see Configuring Controls.

If your driver has read-only controls (EEPROM, OTP or fuse ID), you must implement a function for each control and make imx185_ctrls_init{} call it.

imx185_ctrls_init {

...

err = imx185_fuse_id_setup(priv);

...

}

Device Tree Parser

The imx185_parse_dt() function, which parses device trees, searches the device tree node for settings for clocks, GPIOs, and regulators, as well as any device-specific properties.

The values include but are not limited to:

• mclk

• pwdn-gpios

• reset-gpios

• af-gpios

• avdd-reg

• dvdd-reg

• iovdd-reg

For information about setting up device trees, see Device Properties. You must match the name list with the names in the device tree to get the respective property-value pairs.

Media Controller Setup

A few additional components are needed to set up the media controller for V4L2. The “open” function call is a placeholder operation for satisfying the V4L2 subdevice internal operation requirements. The setup likely does not need to be changed apart from changing the name to match the devices.

imx185_open()

Subdevice function operations must link to the open operation:

static const struct v4l2_subdev_internal_ops

imx185_subdev_internal_ops = {

.open = imx185_open,

};

Media entity function operations must link to the V4L2 subdevice link validation method:

static const struct media_entity_operations imx185_media_ops = {

.link_validate = v4l2_subdev_link_validate,

};

Sensor Driver Probing

This topic describes the probing functions of the sensor driver.

Entry Point for Initialization During Boot

The imx185_probe() function is the driver’s entry point. The function first allocates memory for common data and sensor-private data. For more information, see Sensor-Private Data.

common_data = devm_kzalloc(&client->dev,

sizeof(struct camera_common_data), GFP_KERNEL);

priv = devm_kzalloc(&client->dev,

sizeof(struct imx185) + sizeof(struct v4l2_ctrl *) *

ARRAY_SIZE(ctrl_config_list),

GFP_KERNEL);

It then initializes regmap.

priv->regmap = devm_regmap_init_i2c(client, &sensor_regmap_config);

Then it calls the other initialization functions.

// See the section Parser for Device Trees: stored in private

// data pdata field.

imx185_parse_dt(client);

// See Power Functions.

imx185_power_get(priv);

// Link the appropriate subdevice ops. See

// Setting Up V4L2 Subdevice and Camera Common.

v4l2_i2c_subdev_init(&common_data->subdev, client, &subdev_ops);

// See Control Initialization.

imx185_ctrls_init(priv);

Next, the function links the common data and sensor-private values:

// Link to ops. See Setting Up V4L2 Subdevice and Camera Common.

common_data->ops = & imx185_common_ops;

// Control handler linking.

common_data->ctrl_handler = &priv->ctrl_handler;

// I2C client passed in to probe.

common_data->i2c_client = client;

// Default to frame format 0. See Mode Tables.

common_data->frmfmt = & imx185_frmfmt[0];

// Based on default data format definition, generally defaults

// to the same format. See Macro Definitions.

common_data->colorfmt = camera_common_find_datafmt(

IMX185_DEFAULT_DATAFMT);

// Power-handler linking.

common_data->power = &priv->power;

// Sensor-private data linking.

common_data->priv = (void *)priv;

// Number of format is frame format. See Macro Definitions.

common_data->numfmts = ARRAY_SIZE(imx185_frmfmt);

// Initialize camera common code.

camera_common_initialize(common_data, “imx185”);

Setup of Default Values for Common Data

For more information, see Macro Definitions.

common_data->def_mode = IMX185_DEFAULT_MODE;

common_data->def_width = IMX185_DEFAULT_WIDTH;

common_data->def_height = IMX185_DEFAULT_HEIGHT;

common_data->def_clk_freq = IMX185_DEFAULT_CLK_FREQ;

// I2C client passed in to probe

priv->i2c_client = client;

// Link to common data above.

priv->s_data = common_data;

// Link to V4L2 subdevice handler.

priv->subdev = &common_data->subdev;

// Link subdevice dev to i2c_client dev (for media controller usage).

priv->subdev->dev = &client->dev;

Setup of Media Controller

All imager devices must register themselves as sub-devices to the media controller framework. VI acts as the primary device which controls the binding, parses the device tree, and establishes the media links once all of the sub-devices are registered.

Each subdevice registers with the media controller framework by defining the entity and pads information.

• Entity: The unit device represented by the media controller framework for establishing connections. Each media entity can have multiple pads. The framework provides a list of known entity types and corresponding media operations.

• Pad: Determines whether the device behaves as a SINK or a SOURCE port. An imager device must have a SOURCE port which is bound to VI. If an imager device has a SINK port, then the SOURCE port which binds the device SINK port must be represented in the device tree to complete the binding.

// Link subdevice internal and media entity operations.

priv->subdev->internal_ops = & imx185_subdev_internal_ops;

// Set up subdevice flags and media entity type.

priv->subdev->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |

V4L2_SUBDEV_FL_HAS_EVENTS;

// Set up media controller pad flags.

priv->pad.flags = MEDIA_PAD_FL_SOURCE;

priv->subdev->entity.type = MEDIA_ENT_T_V4L2_SUBDEV_SENSOR;

priv->subdev->entity.ops = &imx185_media_ops;

// Initialize and register the subdevice and media entity with

// the media controller framework.

tegra_media_entity_init(&priv->subdev->entity, 1, &priv->pad, true,

true);

v4l2_async_register_subdev(priv->subdev);

Removing Sensor Drivers

The imx185_remove() function is called on device shutdown to release any resources prior to removing the sensor device instance.

// Unregister V4L2 subdevice.

v4l2_async_unregister_subdev(priv->subdev);

// Cleanup and unregister subdevice and media entity with media

// controller framework.

media_entity_cleanup(&priv->subdev->entity);

// See Control Initialization for details on freeing the control

// handler.

v4l2_ctrl_handler free(&priv->ctrl_handler);

// Clean up camera common code.

camera_common_cleanup(s_data);

V4L2 Kernel Driver (Version 2.0)

This topic is based on the Video for Linux 2 (V4L2) driver for the Sony IMX185 sensor, located at:

<top>/kernel/nvidia/drivers/media/i2c/imx185.c

Examine this source file to obtain a complete understanding of the driver.

For information on the V4L2 implementation, consult:

https://linuxtv.org/downloads/v4l-dvb-apis/

Macro Definitions

Sensor-specific macro definitions include:

• Sensor register addresses

• Constant values used for calculating new settings

Do not add control-specific data (minimum, maximum, default value, etc.) as macros. Define them in the sensor device tree .dtsi file.

Sensor-Private Data

This structure contains private data specific to the sensor:

struct imx185 {

struct i2c_client *i2c_client;

struct v4l2_subdev *subdev;

u8 fuse_id[FUSE_ID_SIZE];

u32 frame_length;

s64 last_wdr_et_val;

struct camera_common_data *s_data;

struct tegracam_device *tc_dev;

};

The following table describes the sensor properties.

Property	Value
i2c_client	A handle for i2c_client, used to gain access to the sensor I2C client instance.
subdev	A handle for a V4L2 subdevice, used to perform subdevice operations (ops).
fuse_id	Holds fuse ID information for the sensor in this char array.
frame_length	Holds the previous value of frame_length.
last_wdr_et_val	Holds the previous exposure time for WDR mode.
s_data	Points to struct camera_common_data, which holds information used by all sensor drivers. See camera_common.h for details.
tc_dev	Points to struct tegracam_device, used by Jetson V4L2 Camera Framework to control camera devices. See tegracam_core.h for details.

Regmap

Initialize the values of the regmap_config structure’s reg_bits and val_bits members according to the I2C interface of the sensor:

struct regmap_config sensor_regmap_config {

reg_bits = 16;

val_bits = 8;

};

reg_bits holds the size of the register address, in bits. val_bits holds the size of the register value, in bits. Check your sensors vendor’s register programming table to determine the proper values.

Sensor Controls

The following table displays the custom V4L2 controls supported by Jetson V4L2 Camera Framework.

Property	Value
TEGRA_CAMERA_CID_GAIN	Sensor gain, a 64-bit value in Q format. The value is n in a 1:n ratio. For example, a value of 3 represents a ratio of 1:3, yielding 3x amplification. The gain_factor property described in Property-Value Pairs (Version 2.0 Only) controls the Q format and the range of the control. For more information, see Gain Control.
TEGRA_CAMERA_CID_FRAME_RATE	Frame rate, in frames/second.
TEGRA_CAMERA_CID_EXPOSURE	Exposure time, in seconds. For more information, see Exposure Control.
TEGRA_CAMERA_CID_EXPOSURE_SHORT*	Exposure time, in seconds. Used for the short exposure frame for HDR sensors. For more information, see Exposure Control.
TEGRA_CAMERA_CID_HDR_ENABLE*	A Boolean value; indicates whether the sensor is running in HDR sensor mode (true) or normal sensor mode (false).
TEGRA_CAMERA_CID_EEPROM_DATA	Reads the module’s EEPROM data.
TEGRA_CAMERA_CID_OTP_DATA*	Reads the module’s One-Time Programmable (OTP) memory data.
TEGRA_CAMERA_CID_FUSE_ID*	Reads the sensor fuse ID.
TEGRA_CAMERA_CID_GROUP_HOLD‡	A Boolean value. Sensor group hold (register hold) control for sensors that support this feature.
TEGRA_CAMERA_CID_SENSOR_MODE_ID*	Forces Jetson V4L2 Camera Framework to use a specific sensor mode, bypassing its default mode selection logic. Value is a zero-based index, the sensor mode index specified in the sensor device tree .dtsi file.
TEGRA_CAMERA_CID_VI_BYPASS_MODE†	A Boolean value. Set to true to bypass VI settings in the kernel driver. Default value is true when using the camera core library.
TEGRA_CAMERA_CID_OVERRIDE_ENABLE†	Overrides gain and exposure settings from the default sensor mode table.
* Optional. † Provided by the NVIDIA V4L2 Media Controller Framework kernel driver. ‡ Provided by the Jetson V4L2 Camera Framework.

Note:

To define additional controls you must change the camera core library.

Exposure Controls

Sensor exposure is controlled by the exposure and frame rate settings the TEGRA_CAMERA_CID_EXPOSURE, TEGRA_CAMERA_CID_EXPOSURE_SHORT, and TEGRA_CAMERA_CID_FRAME_RATE controls. The TEGRA_CAMERA_CID_EXPOSURE_SHORT control is optional, and may only be specified for HDR sensors.

All of these controls have 64‑bit exposure values that are unprocessed except for format conversion; see Fixed Point Format.

See the imx185_set_frame_rate() and imx185_set_exposure() functions in the file imx185.c.

Fixed Point Format

The driver receives control values for the TEGRA_CAMERA_CID_GAIN, TEGRA_CAMERA_CID_FRAME_RATE, TEGRA_CAMERA_CID_EXPOSURE, and TEGRA_CAMERA_CID_EXPOSURE_SHORT controls as fixed point (Q format) values generated using the factor and step properties from the device tree .dtsi file. It uses fixed point format to avoid using floating point values in the kernel. The driver must convert the fixed point format values back to their original precisions during final calculation.

Control Handlers

Control handlers are the actual implementations of the controls. There are two types of control handlers in JetsonV4L2 Camera Framework: setter control handlers and fill-string control handlers.

Setter Control Handlers (for Writing Settings)

Setter control handlers are called by the V4L2 s_ctrl() function. They update the sensor settings during sensor operation. The setter control handlers are:

• imx185_set_group_hold()

• imx185_set_gain()

• imx185_set_framerate()

• imx185_set_exposure()

For a version 2.0 driver, the TEGRA_CAMERA_CID_GROUP_HOLD control is required. Therefore, each driver must implement the handler for group hold control. For sensors that do not support hardware group hold functionality, create a dummy group hold handler.

Fill-String Control Handlers (for Reading Settings)

Fill-string controls report static sensor information like EEPROM/OTP data and fuse ID.

All read-only driver-specific controls must have an entry to provide valid data for user space requests. This routine acquires the static data from the sensor driver during initialization and assigns it to the controls.

imx185_fill_string_ctrl {

...

switch (ctrl->id) {

case TEGRA_CAMERA_CID_FUSE_ID:

...

}

...

}

How Controls Are Implemented

You implement sensor controls by registering to Jetson V4L2 Camera Framework and then providing the link to the control handlers.

Registering the Controls

static const u32 ctrl_cid_list[] = {

TEGRA_CAMERA_CID_GAIN,

TEGRA_CAMERA_CID_EXPOSURE,

TEGRA_CAMERA_CID_FRAME_RATE,

TEGRA_CAMERA_CID_GROUP_HOLD,

TEGRA_CAMERA_CID_FUSE_ID,

TEGRA_CAMERA_CID_HDR_EN,

TEGRA_CAMERA_CID_SENSOR_MODE_ID,

};

How to provide the link to the control handlers

1. Define a tegracam_ctrl_ops structure.

1. Set the numctrls member to the number of controls provided by the driver.

2. Set the ctrl_cid_list member to point to the ctrl_cid_list[] structure.

2. Link each setter control to the handler.

static struct tegracam_ctrl_ops imx185_ctrl_ops = {

.set_gain = imx185_set_gain,

.set_exposure = imx185_set_exposure,

.set_frame_rate = imx185_set_frame_rate,

.set_group_hold = imx185_set_group_hold,

};

3. For fill-string controls (EEPROM, OTP, Fuse ID), provide the size of the string data and link the control to the handler.

static struct tegracam_ctrl_ops imx185_ctrl_ops = {

. . .

.string_ctrl_size = {0, IMX185_FUSE_ID_STR_SIZE},

.fill_string_ctrl = imx185_fill_string_ctrl,

};

Jetson V4L2 Camera Framework supports up to eight fill-string controls. The first three string controls are reserved for reading EEPROM data, fuse ID, and OTP data, respectively.

Setting Up Registers for the Control Handler

Control handlers update sensor settings by programming related registers. The functions below generate the proper register address/value pair to be used later by register read/write functions.

• imx185_get_frame_length_regs()

• imx185_get_coarse_time_regs_shs1()

• imx185_get_coarse_time_regs_shs2()

• imx185_get_gain_reg()

Read-Write Wrapper in the Register

The following functions are wrappers for the read-write interface of the I2C register. For IMX185, use the regmap interface. You may modify these functions for other interfaces.

• imx185_read_reg()

• imx185_write_reg()

• imx185_write_table()

Power Functions

Power functions control sensor power states.

Controls	Operation
imx185_power_on	Performs the power-on sequence. You must modify this function according to the specification sheets. The function calls regulator_enable() to turn on regulators, and gpio_set_value() to toggle the GPIO pins.
imx185_power_off	Performs the power-off sequence. You must modify this function according to the specification sheets. The function calls regulator_enable() to turn on regulators, and gpio_set_value() to toggle the GPIO pins.
imx185_power_put	Calls regulator_put() on all regulators.
imx185_power_get	Calls the camera_common_regulator_get() helper function or the regulator_get() function to acquire all regulators.

The driver must call power_get() to acquire all regulators before it calls power_on() and power_off(). It must call power_put() to release all regulators when the driver is released.

Stream Functions

Stream functions control sensor streaming states.

Controls	Operation
imx185_set_mode	Sets the sensor to a specific resolution and format. The function must program all of the registers with values specified in the mode tables. For details of the mode tables, See Mode Tables.
imx185_start_streaming	Starts sensor streaming by enabling the stream control bits in the registers. This function can also enable the sensor test pattern by setting the necessary registers.
imx185_stop_streaming	Stops sensor streaming by disabling the stream control bits in registers.

Miscellaneous Functions

Controls	Operation
imx185_parse_dt	Performs device tree parsing. Called during driver initialization to read hardware information from the device tree .dtsi file.
imx185_board_setup	Optional. Called during driver initialization to perform critical one-time tasks that are not performed during normal sensor operations. Examples are reading the EEPROM/fuse ID and configuring other peripherals that are needed for sensor operations.

Control Operations

Control operations (ops) hold pointers to functions (handlers) defined by the driver, which perform specific operations on the sensor.

V4L2 compliant sensor drivers use many control operations, described below.

V4L2 Control Ops

V4L2 control ops communicate with the V4L2 framework. These operations are handled automatically in the Jetson V4L2 Camera Framework, and need not be added to individual sensor driver:

• v4l2_ctrl_ops

• v4l2_subdev_ops

• v4l2_subdev_video_ops

• v4l2_subdev_core_ops

• v4l2_subdev_pad_ops

Jetson V4L2 Camera Framework Ops

Jetson V4L2 Camera Framework ops update sensor settings during camera operation. For details of each control, see How Controls Are Implemented.

static struct tegracam_ctrl_ops imx185_ctrl_ops = {

// number of controls and control table

.numctrls = ARRAY_SIZE(ctrl_cid_list),

.ctrl_cid_list = ctrl_cid_list,

// Setter controls

.set_gain = imx185_set_gain,

.set_exposure = imx185_set_exposure,

.set_frame_rate = imx185_set_frame_rate,

.set_group_hold = imx185_set_group_hold,

// Fill-String controls

.string_ctrl_size = {0, IMX185_FUSE_ID_STR_SIZE},

.fill_string_ctrl = imx185_fill_string_ctrl,

};

Camera Common Ops

Camera Common ops are ops that must be implemented for each sensor driver. They are the basic functions which each sensor driver uses.

static struct camera_common_sensor_ops imx185_common_ops = {

// number of formats and format table

.numfrmfmts = ARRAY_SIZE(imx185_frmfmt),

.frmfmt_table = imx185_frmfmt,

// power controls

.power_on = imx185_power_on,

.power_off = imx185_power_off,

.power_get = imx185_power_get,

.power_put = imx185_power_put,

// register read/write control

.write_reg = imx185_write_reg,

.read_reg = imx185_read_reg,

// device tree parsing

.parse_dt = imx185_parse_dt,

// Sensor streaming controls

.set_mode = imx185_set_mode,

.start_streaming = imx185_start_streaming,

.stop_streaming = imx185_stop_streaming,

};

v4l2_subdev_internal_ops

V4L2 subdevice ops are used by V4L2 Framework to start the driver.

static const struct v4l2_subdev_internal_ops

imx185_subdev_internal_ops = {

.open = imx185_open,

};

Boot-Time Initialization

This section describes how the sensor driver is instantiated at kernel boot time. If all steps are performed and no errors are reported, the driver creates a new V4L2 device tree node for each sensor, and the sensors are ready for use.

Control Initialization

The IMX185 gives access to ctrl_cid_list[] while registering the sensor driver. Jetson V4L2 Camera Framework iterates through the entries and sets up the controls required for the driver.

The framework sets up the controls with properties specified in the device tree .dtsi file.

Device Tree Parser

The imx185_parse_dt() function parses the device tree .dtsi file, gets the device tree node, and looks for the parameters required by the sensor driver, according to the sensor-related private data.

The required parameters include, but are not limited to:

• mclk

• pwdn-gpios

• reset-gpios

• af-gpios

• avdd-reg

• dvdd-reg

• iovdd-reg

For information about how to set up device trees, see Device Properties. You must match the name list with the names in the device tree to get the respective property-value pairs.

Sensor-Driver Probing

The imx185_probe() function is the driver’s entry point. It first allocates memory for sensor-private data and the tegracam_device structure, which Jetson V4L2 Camera Framework uses.

priv = devm_kzalloc(&client->dev,

sizeof(struct imx185), GFP_KERNEL);

tc_dev = devm_kzalloc(&client->dev,

sizeof(struct tegracam_device), GFP_KERNEL);

The function then initializes the context needed for Jetson V4L2 Camera Framework registration:

// I2C client passed in to probe.

tc_dev->i2c_client = priv->i2c_client = client;

// Set the name for the tegra camera device (sensor name).

strncpy(tc_dev->name, "imx185", sizeof(tc_dev->name));

// Initialize regmap configuration.

tc_dev->dev_regmap_config = &sensor_regmap_config;

// Initialize sensor common ops and tegra camera control ops.

tc_dev->sensor_ops = &imx185_common_ops;

tc_dev->tcctrl_ops = &imx185_ctrl_ops;

It then registers the device as a Tegra camera device using Jetson V4L2 Camera Framework.

// Register device to initialize the framework context.

// and v4l2 subdevice information.

tegracam_device_register(tc_dev);

// Set sensor data as private data of tc_dev.

tegracam_set_privdata(tc_dev, (void *)priv);

// Run board setup to ensure the sensor hardware is accessible.

imx185_board_setup();

Finally it registers the device as a V4L2 subdevice.

// Once setup is successful, register the driver with V4L2 framework.

tegracam_v4l2subdev_register(tc_dev, true);

Removing Sensor Drivers

The imx185_remove() function removes the sensor device instance. The framework calls it on device shutdown.

The functions below unregister the sensor as a V4L2 subdevice and unregister the driver from Jetson V4L2 Camera Framework.

tegracam_v4l2subdev_unregister(priv->tc_dev);

tegracam_device_unregister(priv->tc_dev);

Loadable Kernel Module (LKM)

A loadable kernel module (LKM) is a mechanism for adding code to, or removing code from, the Linux kernel at run time.

A sensor driver that uses the Jetson V4L2 Camera Framework can now be built as a loadable kernel module. The sensor driver need not be changed.

LKM is very helpful during sensor driver development, especially for debugging. Rather than flash a new kernel image to the Jetson device, you can simply unload the driver, update and rebuild the driver, and load the rebuilt driver back into the Linux kernel.

To configure a sensor driver as a loadable module

1. Place this setting in the kernel configuration file, replacing the previous CONFIG_VIDEO setting for the camera (if any):

CONFIG_VIDEO <camera>=m

Where <camera> is the name of the camera, e.g. imx185.

2. Rebuild the kernel and the sensor driver.

When the sensor driver has been rebuilt, you may find it at:

<OUT>/nvidia/kernel/drivers/media/i2c/<camera>.ko

Now you can copy the sensor driver to the Jetson device’s file system and load it, as described in To load a new sensor module.

Alternatively, you can make the Linux kernel load your sensor drover automatically by rebuilding the kernel image and flashing the Jetson device.

To load a new sensor module

1. Copy the sensor driver file (<camera>.ko, built by To configure a sensor driver as a loadable module) to the Jetson device’s filesystem.

2. To unload the old sensor module, enter this command:

sudo rmmod <sensor_LKM_module_basename>

(The basename is the filename without the .ko extension.)

3. To load the new sensor module, enter this command:

sudo insmod <path_to_sensor_drive_file>

Kernel Configuration

When you have finished developing the driver, add the new device information to the kernel configuration file so it can be enabled for the kernel, as follows:

CONFIG_VIDEO <camera>=y
or
CONFIG_VIDEO_<camera>=m (for a loadable kernel module)

Where <camera> is the name of the camera, e.g. imx185.

The kernel configuration file is at:

<top>/kernel/kernel-4.9/arch/arm64/configs/tegra_defconfig

Device Registration

After driver development is complete, you must also add the new device information to the system kernel device tree so it can be registered (instantiated) when the kernel boots. There are two ways of registering your device.

Using Plugin Manager

If your camera module has on-board EEPROM and is programmed with a valid camera ID, you can use Plugin Manager to register it. If your device module does not meet these requirements, use the main platform device tree instead by following the directions in Using the Main Platform Device Tree File.

Plugin Manager automatically links devices when the system kernel boots. Plugin Manager uses the camera module ID returned by Bootloader to update the device tree entries with proper information at runtime. Plugin Manager allows a single system image to support multiple camera devices. To change camera modules, power down the device, replace the camera module, and then reboot. The new module works automatically.

Prerequisites

• You have obtained the kernel source files. For more information, see Synchronizing the Kernel.

To use Plugin Manager support

1. Add your .dtsi file to the camera configuration .dtsi file.

2. Set the status of your device tree nodes to “disabled”:

imx185_cam0: imx185_a@1a {

status = "disabled";

};

3. Add the override information to:

<top>/hardware/nvidia/platform/t19x/common/kernel-dts/t19x-common-modules/tegra194-camera-plugin-manager.dtsi

4. Update the camera Plugin Manager .dtsi with proper override information.

fragment-imx185@0 {

ids = "LPRD-002001";

override@0 {

target = <&imx185_cam0>;

_overlay_ {

status = "okay";

};

. . .

};

If a specific camera board is found when the kernel boots, the override data is applied to the specific device tree nodes, and they are made available for the system.