Jetson Virtual Channel with GMSL Camera Framework

Applies to: Jetson AGX Xavier series and Jetson TX2 series
This topic contains details about:
The Gigabit Multimedia Serial Link (GMSL) protocol
Hardware connectivity for the serializer/deserializer in the reference module (see Platforms)
The software framework
Configuration, including virtual channel programming
NVIDIA validates the reference module on NVIDIA® Jetson™ TX2 and NVIDIA® Jetson AGX Xavier™ platforms with Sony IMX390 sensors as source. However, the reference module can be used as a starting point for bringing up another GMSL module on Jetson TX2 and Jetson AGX Xavier.
The software framework described in this document can be used as reference for developing SerDes (Serializer-Deserializer) links other than GMSL.
Note:
The reference GMSL module described here uses the CSI interface. No other interface is validated.

Platforms

GMSL and other aggregators are not supported on NVIDIA® Jetson Nano™ devices and NVIDIA® Jetson™ TX1 platforms, as they do not support virtual channels. This document applies to Jetson AGX Xavier series and Jetson TX2 platforms only.
The reference GMSL setup is:
Sensor: Sony dual IMX390, RAW12/1080p/30fps, CSI port A, x2 lanes
Serializer: Maxim MAX9295
Deserializer: Maxim MAX9296

GMSL Protocol

Maxim Integrated supports GMSL as a communication link for video applications in the automotive industry. GMSL is based on SerDes (Serializer-Deserializer) technology; that is, it uses a serializer on the transmitting side and a deserializer on the receiving side. The following diagram shows the high-level architecture of GMSL.
GMSL is specifically designed for use in Advanced Driver Assistance Systems (ADAS) and Camera Monitoring Systems (CMS). It can provide video transfer speeds up to 6 GB/second. It uses STP or coaxial cable, which are both inexpensive and very robust for EMC disturbances.

GMSL Camera

The following diagram shows the control, data, and clock connections for the reference GMSL module validated on Jetson AGX Xavier and Jetson TX2.
In this GMSL setup, two sensors are paired with their respective serializers, each streaming 1080p/30fps RAW12 pixels over x2 CSI MIPI lanes.
The serializers are connected to a GMSL deserializer device through different GMLS ports (ports A and B) and GMSL links using coaxial cables.
On the output port side, the deserializer is connected to a Jetson TX2 or Jetson AGX Xavier SoC on the desired CSI port (port A in this example).
To transmit two different pixel streams from two sensors to a receiver at a shared CSI port, the deserializer assigns a unique virtual channel ID to each stream. The virtual channel ID is software-configurable via the device tree. It must match the stream’s virtual channel ID programmed on the receiver side in the NVIDIA SoC.
This reference GMSL setup uses a 2x (x2/x4/x1) CSI deserializer. It can host up to two sensors via serializers.
To host four sensors over single shared CSI port you must use a 4x aggregator, as shown in the following diagram:

CSI Connectivity

The table below shows the maximum number of sensor connections supported on each Jetson platform.
Configuration
Maximum Connections Supported
Jetson TX2 series
Jetson AGX Xavier series
No aggregator
6
6
Aggregator with ISP
12
16
Aggregator without ISP
12
24

Jetson TX2

The maximum number of virtual channels supported on Jetson TX2 is 12 via three 4x aggregators connected to each of the CSI bricks in x4, x2, or x1 lane configuration (of any valid combination of them).
The diagram above shows Jetson TX2 connections for 12 cameras with virtual channels.

Jetson AGX Xavier Series

Jetson AGX Xavier series supports a maximum of 16 virtual channels with ISP, or 24 virtual channels without ISP.
The diagram above shows sensor connections to each of the NVIDIA® Xavier™ CSI bricks (a total of 4 CSI bricks) in x4 or /x2 or /x1 possible lane configuration.
The diagram above shows four sensors connected to each port of CSI bricks AB and CD in x2 or x1 lane configuration, and four sensors connected to each of the remaining CSI bricks EF and GH in x4, x2, or x1 lane configuration.
The CSI aggregator uses virtual channels to connect to four cameras over one CSI connection.
The Jetson TX2 or Jetson AGX Xavier VI muxer sends each camera frame to a different location in memory.
In software, each camera appears as a separate V4L2 device.

Hardware Module Connectivity

The figure below shows how two sensors are connected to a Jetson AGX Xavier series device using a GMSL reference module and a MAX9295/MAX9296 SerDes. Two sensors may be connected to a Jetson TX2 the same way.
The GMSL MAX9296 deserializer is connected to the Jetson platform via a MIPI adapter and MIPI white cables, which are plugged into the platform’s camera connector socket. This is one of several ways the aggregator hardware module can be connected to the Jetson platform.
ftegra-
 

Software Framework and Programming

This section describes the kernel drivers and device tree programming required for GMSL and virtual channel.
The following diagram shows the high-level architecture of the kernel drivers and devices.

Driver Framework

As shown in the diagram above, there are separate kernel drivers for serializer and deserializer devices, apart from the sensor driver. In the reference module they are the MAX9295 serializer and MAX9296 deserializer drivers.
In this framework, the sensor driver is exposed to the rest of the system as would be any other generic V4L2 sensor driver without external aggregator. All of the SerDes programming happens “under the hood” through the sensor driver.
The SerDes kernel drivers are not registered as client programmable and isolated devices V4L2 or any other sensor framework.
The reason for this design is that a certain fixed sequence of operations must be performed in the SerDes, and it does not quite fit the generic V4L2 framework sequence for sensors.
SerDes drivers are separate, though. They can be statically linked to any sensors in the device tree, depending on hardware connectivity, but they are controlled by the sensor driver only. They do not expose any programmable functionality directly to user clients.
A sensor driver internally links to the SerDes drivers to perform device and stream control operations such as power on/off, control setup/release, stream setup/release, and stream start/stop, based on device tree configuration.
Driver Programming
The serializer/deserializer drivers control the serializer/deserializer modules through the serializer/deserializer API, described in Jetson Linux Drive Package API References.
The structure gmsl_link_ctx is the core entity which defines the entire link configuration from sensor to serializer link to deserializer link for each sensor source. It is documented in the API References.
Most of the structure fields are populated by the sensor driver from the sensor’s gmsl-link node in the device tree during device boot. (See Module Device Tree for node details.) Some of the fields are populated by the serializer and deserializer drivers.
The sensor driver populates an instance of the structure and passes it to the serializer and deserializer drivers, which make use of the configuration details found in the structure context during power-on, control pipeline setup, and data streaming pipeline setup calls.
As defined in this structure, the sensor and its corresponding serializer device each have a physical I2C slave address, assigned according to the device data sheet, and a proxy I2C slave address, which is user-defined.
The reason for this is that all of the devices sharing the same hardware connection in the GMSL setup must be identified to the I2C bus with unique physical I2C slave addresses. The physical I2C slave address of each device is fixed, and is the same for similar types of devices, such as all sensor devices of a single make and model that are assigned to same slave. This causes address conflicts. To resolve the conflicts the driver uses a different proxy I2C slave address for each such device. The proxy I2C slave addresses are set statically in the device tree, and are assigned during device boot.
The following diagram shows the call flow among the GMSL kernel drivers during device boot.
Graphical user interface Description automatically generated
Power management is handled by the shared deserializer driver instead of the sensor driver because each sensor device’s driver is unaware of other sensor devices in the GMSL module setup, but the deserializer device is common among all of the sensors connected to it.
The stream-on and stream-off calls are mapped directly to the serializer and deserializer drivers.
Stream setup is required for both SerDes drivers, whereas stream-on and stream-off are controlled by the deserializer driver only.
The following figure shows the call flow among the GMSL kernel drivers for a stream-on or stream-off operation.
Graphical user interface, diagram Description automatically generated

Device Tree Programming

The device tree describes the hardware connections to the system as well as the static device properties of the drivers.
Platform Device Tree
Hardware connections and device addressing are configured in the platform device tree.
As explained in Driver Programming, each sensor and serializer device has two addresses: a physical I2C slave address and a proxy I2C slave address. Both are defined in the platform device tree.
In the reference GMSL module both sensors have the physical I2C slave address 0x1a, but must be assigned distinct proxy addresses. The proxy I2C slave addresses are user-configurable, by default 0x1b and 0x1c.
Similarly, for serializer devices the physical I2C slave address is 0x62, and the proxy I2C slave addresses are user-configurable, by default 0x40 and 0x60.
The following table shows the I2C slave address assignments for the reference GMSL setup.
 
I2C Slave Address Assignments
Sensor1
Sensor2
Ser1
Ser2
Des
Physical I2C slave address
0x1a
0x1a
0x62
0x62
0x48
Proxy I2C slave address
0x1b
0x1c
0x40
0x60
NA
Each sensor device initiates a power_on request to the deserializer driver to perform GMSL link configuration and power on the shared deserializer device. In this power-on call, the deserializer driver only enables the GMSL link to which the caller’s sensor device is connected, and keeps other one disabled, so that there will be no address conflict between the two links. It then updates all devices’ proxy I2C slave addresses on the active link to the ones defined in the device tree. It follows the same procedure for the other GMSL link when the deserializer driver gets a power_on request from the sensor device connected to the other GMSL link. In this way, the deserializer driver updates all of the proxy I2C save addresses of all devices hosted by the deserializer device.
For example, in the reference GMSL setup, when the deserializer gets a power-on request from the sensor device connected to link A, it enables GMSL link A and disables GMSL link B. It first communicates to the serializer and the sensor devices on link A at their respective physical I2C slave addresses (0x1a for sensor1 and 0x62 for Ser1), and then updates those physical I2C slave addresses to proxy I2C slave addresses (0x1b for sensor1 and 0x40 for Ser1). Then it performs the same steps for GMSL link B when the sensor device on GMSL link B requests power_on of the deserializer driver. Finally, it sets up the GMSL link in dual mode. Thus, each device is identified uniquely over the I2C bus. All further device communication takes place on the proxy I2C slave addresses until reboot.
The GMSL link addresses are currently programmed at boot time, during probing, due to control configuration timing constraints.
The following code is an example of platform device configuration for the GMSL setup shown in the diagrams above. For full details, see the platform device tree file:
For Jetson TX2: tegra186-quill-camera-imx390-a00.dtsi
For Jetson AGX Xavier series: tegra194-p2822-0000-camera-imx390-a00.dtsi
tca9546@70 {
/* The deserializer is connected to Tegra Jetson camera connector port via MIPI adapter over pca9546 i2c expander, so all the GMSL device nodes go under here. */
compatible = "nxp,pca9546";
i2c@0 {
/* As in above setup, deserializer (which hosts all the serializers and sensors) is connected to adaptor onto port A, so on i2c expander all the GMSL devices go under i2c@0, the expander assigns 0x30
unique address to port 0.*/
reg = <0>;
dser: max9296@48 {
/* single common deserializer at 0x48 */
compatible = "nvidia,max9296";
reg = <0x48>;
csi-mode = "2x4"; /* this tells max CSI lane configuration */
max-src = <2>; /* max sources */
/* As deserializer is common among sensor devices, the reset and power rails are controlled via deserializer */
reset-gpios = <&tegra_main_gpio CAM0_RST_L GPIO_ACTIVE_HIGH>;
vdd_cam_1v2-supply = <&en_vdd_cam_1v2>;
};
 
ser_prim: max9295_prim@62 { /* Default serializer device */
compatible = "nvidia,max9295";
reg = <0x62>;
is-prim-ser; /* primary ser device */
};
 
ser_a: max9295_a@40 {
/* serializer device connected at link A at proxy address 0x40 (the proxy address are assigned runtime) */
compatible = "nvidia,max9295";
reg = <0x40>;
nvidia,gmsl-dser-device = <&dser>; /* link to its deserializer device */
};
 
ser_b: max9295_b@60 {
/* serializer device connected at link A at proxy address 0x60 (the proxy address is assigned runtime) */
compatible = "nvidia,max9295";
reg = <0x60>;
nvidia,gmsl-dser-device = <&dser>; /* link to its deserializer device */
};
 
imx390_a@1b {
/* sensor device connected at link A at proxy address 0x1b (the proxy address is assigned runtime). */
def-addr = <0x1a>; /* default slave address is 0x1a */
nvidia,gmsl-ser-device = <&ser_a>; /* link to its serializer device. */
nvidia,gmsl-dser-device = <&dser>; /* link to its deserializer device. */
};
 
imx390_b@1c {
/* sensor device connected at link A at proxy address 0x1b (the proxy address are assigned runtime). */
def-addr = <0x1a>; /* Default slave address is 0x1a. */
/* Define clocks, io pins, power sources */
nvidia,gmsl-ser-device = <&ser_b>; /* Link to its serializer device. */
nvidia,gmsl-dser-device = <&dser>; /* Link to its deserializer device. */
};
}; /* i2c@0 closing */
}; /* tca9546@70 closing */
Module Device Tree
After platform device configuration, the device module configuration must be added to the module device tree. For full details, see:
For Jetson TX2: tegra186-camera-imx390-a00.dtsi
For Jetson AGX Xavier seroes: tegra194-camera-imx390-a00.dtsi
A few snippets are shown here to illustrate the configuration. The virtual channels are set in the module device tree.
The sensor device acts as controller device in the GMSL software framework, so in the module device tree each sensor device node contains a gmsl-link device node which describes the properties of the GMSL link to which the sensor is connected. In the example below, the gmsl-link node describes the sensor connected at GMSL link A.
gmsl-link {
src-csi-port = "b"; /* Port at which sensor is connected to
its serializer device. */
dst-csi-port = "a"; /* Destination CSI port on the Jetson
side, connected at deserializer. */
serdes-csi-link = "a"; /* GMSL link sensor/serializer connected */
csi-mode = "1x4"; /* to sensor CSI mode. */
st-vc = <0>; /* Sensor source default VC ID: 0 unless
overridden by sensor. */
vc-id = <0>; /* Destination VC ID, assigned to sensor
stream by deserializer. */
num-lanes = <2>; /* Number of CSI lanes used. */
streams = "ued-u1", "raw12";
/* Types of streams sensor is streaming. */
};
For any GMSL module configuration, all the fields shown above must be set based on the hardware connectivity.
Virtual Channel Programming in the Module Device Tree
You must add the same vc-id property value to the respective CSI and VI channel nodes in the same module device tree file. This property is used for V4L2 camera use cases.
You must set the vc-id property in the gmsl-link node for each sensor to match the vc_id property in the sensor mode device node.
The vc_id property in sensor mode device tree nodes is used for use cases based on the ARGUS camera. For example:
mode0 { /*mode IMX390_MODE_1920X1080_CROP_30FPS*/
mclk_khz = "24000";
num_lanes = "2";
tegra_sinterface = "serial_a";
vc_id = "0";
VI and CSI Channel Nodes in the Module Device Tree
The module device tree for the reference GMSL setup also performs the port binding to VI and CSI. The number of VI and CSI channels is 2, as in any other dual-sensor setup, but the port-index field would be set based on the hardware connectivity.
vi@15700000 {
num-channels = <2>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
imx390_vi_in0: endpoint {
vc-id = <0>;
port-index = <0>;
bus-width = <2>;
remote-endpoint = <&imx390_csi_out0>;
};
};
port@1 {
reg = <1>;
imx390_vi_in1: endpoint {
vc-id = <1>;
port-index = <0>;
bus-width = <2>;
remote-endpoint = <&imx390_csi_out1>;
};
};
};
};
 
nvcsi@150c0000 {
num-channels = <2>;
#address-cells = <1>;
#size-cells = <0>;
channel@0 {
reg = <0>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
imx390_csi_in0: endpoint@0 {
port-index = <0>;
bus-width = <2>;
remote-endpoint = <&imx390_imx390_out0>;
};
};
port@1 {
reg = <1>;
imx390_csi_out0: endpoint@1 {
remote-endpoint = <&imx390_vi_in0>;
}; };
};
};
channel@1 {
reg = <1>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
imx390_csi_in1: endpoint@2 {
port-index = <0>;
bus-width = <2>;
remote-endpoint = <&imx390_imx390_out1>;
};
};
port@1 {
reg = <1>;
imx390_csi_out1: endpoint@3 {
remote-endpoint = <&imx390_vi_in1>;
};
};
};
};
};
In this configuration the port-index property is set to 0 for both the VI channel 0 and channel 1 nodes, so that both sensors use VI stream 0.
For CSI channel 0 and channel 1 nodes too the port-index property is set to 0, meaning that both channels’ sensors are connected to CSI port A.
The VI and CSI channel nodes also both contain a new property, vc-id, used in V4L2 camera use cases as mentioned above. It must match the gmsl-link device node’s vc-id” property.
The bus-width property specifies the number of CSI lanes used.
tegra-camera-platform node configuration in module device tree
The TEGRA-CAMERA-platform device node specifies details of the sensor modules’ configuration. The configuration below is for the reference GMSL module:
tegra-camera-platform {
compatible = "nvidia, tegra-camera-platform";
num_csi_lanes = <2>;
max_lane_speed = <4000000>;
min_bits_per_pixel = <10>;
vi_peak_byte_per_pixel = <2>;
vi_bw_margin_pct = <25>;
isp_peak_byte_per_pixel = <5>;
isp_bw_margin_pct = <25>;
 
modules {
module0 {
badge = "imx390_rear";
position = "rear";
orientation = "1";
drivernode0 {
pcl_id = "v4l2_sensor";
/* Driver v4l2 device name */
devname = "imx390 30-001b";
proc-device-tree = "/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx390_a@1b";
};
};
module1 {
badge = "imx390_front";
position = "front";
orientation = "1";
drivernode0 {
pcl_id = "v4l2_sensor";
/* Driver v4l2 device name */
devname = "imx390 30-001c";
proc-device-tree = "/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx390_b@1c";
};
};
};
};
num_csi_lanes is set to 2, since both sensors are connected to a single CSI port A in x2 lane fashion. This is unlike other dual sensors, where num_csi_lanes is set to the total number of CSI lanes used by both sensors, since they use different CSI ports.
max_lane_speed is set to an appropriate value for two sensors streaming through a single shared port.
The sensor device nodes are defined using sensor proxy I2C slave addresses instead of their physical I2C slave address, and their paths are set in proc-device-tree property. Similar naming is used for the devname property.
For the rest of the configuration, follow the same guidelines as for other sensors.
Refer to the following module device tree files:
For Jetson TX2: <tegra186-camera-imx390-a00.dtsi>
For Jetson AGX Xavier seroes: <tegra194-camera-imx390-a00.dtsi>
Plugin Manager Device Tree
The device tree additions for Plugin Manager are the same as for any other sensor module.
For reference GMSL setup plugin manager support, see the node fragment-imx390@0 in:
For Jetson TX2 series: tegra186-quill-camera-plugin-manager.dtsi
For Jetson AGX Xavier series: tegra194-camera-plugin-manager.dtsi
Camera Modules Device Tree
All the device nodes in the I2C node of the platform’s camera module device tree are defined in the same way as in the platform device tree. As in the reference GMSL setup, they all go to i2c@0.
The camera module device tree is:
For Jetson TX2 seroes: tegra186-quill-camera-modules.dtsi
For Jetson AGX Xavier series: tegra194-p2822-camera-modules.dtsi
Note:
The vc-id property is set to 0 by default for all VI ports in all modules’ device tree files. You must assign correct values to this property in plugin-manager and the sensor-specific module device tree file.

Constraints

Sensor streams connected to same CSI aggregator and sharing a CSI port must use the same lane configuration. The deserializer driver fails to register a stream if its lane configuration does not match other streams that share the same deserializer device.
Apart from that, CSI aggregators generally support multiple identical cameras running with the same frame rate and sensor mode. Review your aggregator data sheet for additional restrictions if you are using a more complicated configuration.

Validation

Dual GMSL sensor streaming (preview/capture) is validated using Argus cameras, sharing CSI port A, and the V4L2 application.
The 2x CSI deserializer/aggregator is validated. The 4x CSI aggregator is not validated, as its hardware module is currently not available.

Known Issues

This section summarizes known issues in the GMSL camera framework.

General Issues

Preview flickers on scene change.
Sensor tuning and image quality are substandard.

Xavier-Specific

Note:
This limitation and rework only apply to older GMSL kits which do not draw on the on-board (DESER) 1.2 V supply.
The 1.2 V power supply needs rework. The MAX9296 deserializer board used in the reference setup does not draw the required on-board 1.2 V power, and the Jetson AGX Xavier Developer Kit carrier board has dropped support for supplying 1.2 V to the camera connector. (Jetson TX2 does have this support.)
The following rework is required on Jetson AGX Xavier for vendor modules that expect 1.2V from the carrier board:
Use spare LDO for the SATA controller, which can take maximum 1.21 V. The Vmin requirement from the IMX390 reference sensor is 1.14 V, and the range is 1.14 V to 1.26 V, so this LDO supply is enough to supply 1.2 V to the Jetson AGX Xavier carrier board’s camera connector.

Plugin Manager Board ID

The reference GMSL module currently does not have a unique board ID. Use the MIPI adapter’s board ID, LPRD-001. This ID is used by other sensor modules which use similar MIPI adapters, such as IMX274 and IMX185. To work around this issue until it is resolved, the IMX390 module is disabled in these sensor modules; see tegra186-quill-camera-plugin-manager.dtsi.