Jetson Virtual Channel with GMSL Camera Framework

Applies to: Jetson AGX Xavier series

This topic contains details about:

  • The Gigabit Multimedia Serial Link (GMSL) protocol

  • Hardware connectivity for the serializer/deserializer in the reference module (see Reference Setup)

  • The software framework

  • Configuration, including virtual channel programming

NVIDIA validates the reference module on NVIDIA® Jetson AGX Xavier™ series devices with Sony IMX390 sensors as source. However, the reference module can be used as a starting point for bringing up another GMSL module on Jetson AGX Xavier series.

The software framework described in this document can be used as reference for developing SerDes (Serializer-Deserializer) links other than GMSL.

Note

The reference GMSL module described here uses the CSI interface. No other interface is validated.

Reference Setup

The reference GMSL setup is:

  • Sensor: Sony dual IMX390, RAW12/1080p/30fps, CSI port A, x2 lanes

  • Serializer: Maxim MAX9295

  • Deserializer: Maxim MAX9296

GMSL Protocol

Maxim Integrated supports GMSL as a communication link for video applications in the automotive industry. GMSL is based on SerDes (Serializer-Deserializer) technology; that is, it uses a serializer on the transmitting side and a deserializer on the receiving side. The following diagram shows the high-level architecture of GMSL.

GMSL protocol

GMSL is specifically designed for use in Advanced Driver Assistance Systems (ADAS) and Camera Monitoring Systems (CMS). It can provide video transfer speeds up to 6 GB/second. It uses STP or coaxial cable, which are both inexpensive and very robust for EMC disturbances.

GMSL Camera

The following diagram shows the control, data, and clock connections for the reference GMSL module validated on Jetson AGX Xavier series.

Control, data, and clock connections for the reference GMSL module
  • In this GMSL setup, two sensors are paired with their respective serializers, each streaming 1080p/30fps RAW12 pixels over x2 CSI MIPI lanes.

  • The serializers are connected to a GMSL deserializer device through different GMLS ports (ports A and B) and GMSL links using coaxial cables.

  • On the output port side, the deserializer is connected to a Jetson AGX Xavier SoC on the desired CSI port (port A in this example).

  • To transmit two different pixel streams from two sensors to a receiver at a shared CSI port, the deserializer assigns a unique virtual channel ID to each stream. The virtual channel ID is software-configurable via the device tree. It must match the stream’s virtual channel ID programmed on the receiver side in the NVIDIA SoC.

This reference GMSL setup uses a 2x (x2/x4/x1) CSI deserializer. It can host up to two sensors via serializers.

To host four sensors over single shared CSI port you must use a 4x aggregator, as shown in the following diagram:

GMSL setup with 4x aggregator

CSI Connectivity

The table below shows the maximum number of sensor connections supported on each type NVIDIA® Jetson™ device.

Maximum connections supported

Configuration

Jetson AGX Orin

Jetson Xavier NX series

Jetson AGX Xavier series

No aggregator

6

Aggregator with ISP

16

Aggregator without ISP

24

Jetson AGX Orin

Jetson Xavier NX Series

Jetson AGX Xavier Series

Jetson AGX Xavier series supports a maximum of 16 virtual channels with ISP, or 24 virtual channels without ISP.

CSI connectivity for Jetson AGX Xavier series with 16 cameras

The diagram above shows sensor connections to each of the NVIDIA® Xavier CSI bricks (a total of 4 CSI bricks) in x4 or /x2 or /x1 possible lane configuration.

CSI connectivity for Jetson AGX Xavier series with 24 cameras

The diagram above shows four sensors connected to each port of CSI bricks AB and CD in x2 or x1 lane configuration, and four sensors connected to each of the remaining CSI bricks EF and GH in x4, x2, or x1 lane configuration.

The CSI aggregator uses virtual channels to connect to four cameras over one CSI connection.

The Jetson AGX Xavier VI muxer sends each camera frame to a different location in memory.

In software, each camera appears as a separate V4L2 device.

Hardware Module Connectivity

The figure below shows how two sensors are connected to a Jetson AGX Xavier device using a GMSL reference module and a MAX9295/MAX9296 SerDes.

The GMSL MAX9296 deserializer is connected to a Jetson device via a MIPI adapter and MIPI white cables, which are plugged into the device’s camera connector socket. This is one of several ways the aggregator hardware module can be connected to the device.

Two sensor connected to a Jetson device with MAX9295 and MAX9296

Software Framework and Programming

This section describes the kernel drivers and device tree programming required for GMSL and virtual channel.

The following diagram shows the high-level architecture of the kernel drivers and devices.

Kernel drivers and device tree programming

Driver Framework

As shown in the diagram above, there are separate kernel drivers for serializer and deserializer devices, apart from the sensor driver. In the reference module they are the MAX9295 serializer and MAX9296 deserializer drivers.

In this framework, the sensor driver is exposed to the rest of the system as would be any other generic V4L2 sensor driver without external aggregator. All of the SerDes programming happens “under the hood” through the sensor driver.

The SerDes kernel drivers are not registered as client programmable and isolated devices V4L2 or any other sensor framework.

The reason for this design is that a certain fixed sequence of operations must be performed in the SerDes, and it does not quite fit the generic V4L2 framework sequence for sensors.

SerDes drivers are separate, though. They can be statically linked to any sensors in the device tree, depending on hardware connectivity, but they are controlled by the sensor driver only. They do not expose any programmable functionality directly to user clients.

A sensor driver internally links to the SerDes drivers to perform device and stream control operations such as power on/off, control setup/release, stream setup/release, and stream start/stop, based on device tree configuration.

Driver Programming

The serializer/deserializer drivers control the serializer/deserializer modules through the serializer/deserializer API, described in Jetson Linux Drive Package API Reference.

The structure gmsl_link_ctx is the core entity which defines the entire link configuration from sensor to serializer link to deserializer link for each sensor source. It is documented in the API Reference.

Most of the structure fields are populated by the sensor driver from the sensor’s gmsl-link node in the device tree during device boot. (See Module Device Tree for node details.) Some of the fields are populated by the serializer and deserializer drivers.

The sensor driver populates an instance of the structure and passes it to the serializer and deserializer drivers, which make use of the configuration details found in the structure context during power-on, control pipeline setup, and data streaming pipeline setup calls.

As defined in this structure, the sensor and its corresponding serializer device each have a physical I2C slave address, assigned according to the device data sheet, and a proxy I2C slave address, which is user-defined.

The reason for this is that all of the devices sharing the same hardware connection in the GMSL setup must be identified to the I2C bus with unique physical I2C slave addresses. The physical I2C slave address of each device is fixed, and is the same for similar types of devices, such as all sensor devices of a single make and model that are assigned to same slave. This causes address conflicts. To resolve the conflicts the driver uses a different proxy I2C slave address for each such device. The proxy I2C slave addresses are set statically in the device tree, and are assigned during device boot.

The following diagram shows the call flow among the GMSL kernel drivers during device boot.

GMSL kernel driver call flow during boot

Power management is handled by the shared deserializer driver instead of the sensor driver because each sensor device’s driver is unaware of other sensor devices in the GMSL module setup, but the deserializer device is common among all of the sensors connected to it.

The stream-on and stream-off calls are mapped directly to the serializer and deserializer drivers.

Stream setup is required for both SerDes drivers, whereas stream-on and stream-off are controlled by the deserializer driver only.

The following figure shows the call flow among the GMSL kernel drivers for a stream-on or stream-off operation.

GMSL kernel driver call flow for stream-on/stream-off

Device Tree Programming

The device tree describes the hardware connections to the system as well as the static device properties of the drivers.

Platform Device Tree

Hardware connections and device addressing are configured in the platform device tree.

As explained in Driver Programming, each sensor and serializer device has two addresses: a physical I2C slave address and a proxy I2C slave address. Both are defined in the platform device tree.

In the reference GMSL module both sensors have the physical I2C slave address 0x1a, but must be assigned distinct proxy addresses. The proxy I2C slave addresses are user-configurable, by default 0x1b and 0x1c.

Similarly, for serializer devices the physical I2C slave address is 0x62, and the proxy I2C slave addresses are user-configurable, by default 0x40 and 0x60.

The following table shows the I2C slave address assignments for the reference GMSL setup.

I2C Slave Address

Assignments Sensor1

Sensor2

Ser1

Ser2

Des

Physical I2C slave address

0x1a

0x1a

0x62

0x62

0x48

Proxy I2C slave address

0x1b

0x1c

0x40

0x60

NA

Each sensor device initiates a power_on request to the deserializer driver to perform GMSL link configuration and power on the shared deserializer device. In this power-on call, the deserializer driver only enables the GMSL link to which the caller’s sensor device is connected, and keeps other one disabled, so that there will be no address conflict between the two links. It then updates all devices’ proxy I2C slave addresses on the active link to the ones defined in the device tree. It follows the same procedure for the other GMSL link when the deserializer driver gets a power_on request from the sensor device connected to the other GMSL link. In this way, the deserializer driver updates all of the proxy I2C save addresses of all devices hosted by the deserializer device.

For example, in the reference GMSL setup, when the deserializer gets a power-on request from the sensor device connected to link A, it enables GMSL link A and disables GMSL link B. It first communicates to the serializer and the sensor devices on link A at their respective physical I2C slave addresses (0x1a for sensor1 and 0x62 for Ser1), and then updates those physical I2C slave addresses to proxy I2C slave addresses (0x1b for sensor1 and 0x40 for Ser1). Then it performs the same steps for GMSL link B when the sensor device on GMSL link B requests power_on of the deserializer driver. Finally, it sets up the GMSL link in dual mode. Thus, each device is identified uniquely over the I2C bus. All further device communication takes place on the proxy I2C slave addresses until reboot.

The GMSL link addresses are currently programmed at boot time, during probing, due to control configuration timing constraints.

The following code is an example of platform device configuration for the GMSL setup shown in the diagrams above. For full details, see the platform device tree file:

  • For Jetson AGX Xavier series: tegra194-p2822-0000-camera-imx390-a00.dtsi:

    tca9546@70 {
        /* The deserializer is connected to Tegra Jetson camera connector port via
         MIPI adapter over pca9546 i2c expander, so all the GMSL device nodes go
         under here. */
        compatible = "nxp,pca9546";
    
        i2c@0 {
            /* As in above setup, deserializer (which hosts all the serializers and
             sensors) is connected to adaptor onto port A, so on i2c expander all
             the GMSL devices go under i2c@0, the expander assigns 0x30 unique
             address to port 0.*/
            reg = <0>;
    
            dser: max9296@48 {
                /* single common deserializer at 0x48 */
                compatible = "nvidia,max9296";
                reg = <0x48>;
                csi-mode = "2x4";                   /* this tells max CSI lane configuration */
                max-src = <2>;                      /* max sources */
                /* As deserializer is common among sensor devices, the reset and
                 power rails are controlled via deserializer */
                reset-gpios = <&tegra_main_gpio CAM0_RST_L GPIO_ACTIVE_HIGH>;
                vdd_cam_1v2-supply = <&en_vdd_cam_1v2>;
            };
    
            ser_prim: max9295_prim@62 { /* Default serializer device */
                compatible = "nvidia,max9295";
                reg = <0x62>;
                is-prim-ser;                        /* primary ser device */
            };
    
            ser_a: max9295_a@40 {
                /* serializer device connected at link A at proxy address 0x40 (the
                 proxy address are assigned runtime) */
                compatible = "nvidia,max9295";
                reg = <0x40>;
                nvidia,gmsl-dser-device = <&dser>;  /* link to its deserializer device */
            };
    
            ser_b: max9295_b@60 {
                /* serializer device connected at link A at proxy address 0x60 (the
                 proxy address is assigned runtime) */
                compatible = "nvidia,max9295";
                reg = <0x60>;
                nvidia,gmsl-dser-device = <&dser>;  /* link to its deserializer device */
            };
    
            imx390_a@1b {
                /* sensor device connected at link A at proxy address 0x1b (the
                 proxy address is assigned runtime). */
                def-addr = <0x1a>;                  /* default slave address is 0x1a */
                nvidia,gmsl-ser-device = <&ser_a>;  /* link to its serializer device. */
                nvidia,gmsl-dser-device = <&dser>;  /* link to its deserializer device. */
            };
    
            imx390_b@1c {
                /* sensor device connected at link A at proxy address 0x1b (the proxy
                 address are assigned runtime). */
                def-addr = <0x1a>;                  /* Default slave address is 0x1a. */
                /* Define clocks, io pins, power sources */
                nvidia,gmsl-ser-device = <&ser_b>;  /* Link to its serializer device. */
                nvidia,gmsl-dser-device = <&dser>;  /* Link to its deserializer device. */
            };
    
        }; /* i2c@0 closing */
    
    }; /* tca9546@70 closing */
    

Module Device Tree

After platform device configuration, the device module configuration must be added to the module device tree. For full details, see:

  • For Jetson TX: tegra186-camera-imx390-a00.dtsi

  • For Jetson AGX Xavier series: tegra194-camera-imx390-a00.dtsi

A few snippets are shown here to illustrate the configuration. The virtual channels are set in the module device tree.

The sensor device acts as controller device in the GMSL software framework, so in the module device tree each sensor device node contains a gmsl-link device node which describes the properties of the GMSL link to which the sensor is connected. In the example below, the gmsl-link node describes the sensor connected at GMSL link A:

gmsl-link {
    src-csi-port = "b";         /* Port at which sensor is connected to
 its serializer device. */
    dst-csi-port = "a";         /* Destination CSI port on the Jetson side, connected at
                                 deserializer. */
    serdes-csi-link = "a";      /* GMSL link sensor/serializer connected to sensor CSI node. */
    csi-mode = "1x4";
    st-vc = <0>;                /* Sensor source default VC ID: 0 unless overridden by
                                 sensor. */
    vc-id = <0>;                /* Destination VC ID, assigned to sensor stream by
                                 deserializer. */
    num-lanes = <2>;            /* Number of CSI lanes used. */
    streams = "ued-u1", "raw12";
    /* Types of streams sensor is streaming. */
};

For any GMSL module configuration, all the fields shown above must be set based on the hardware connectivity.

Virtual Channel Programming in the Module Device Tree

You must add the same vc-id property value to the respective CSI and VI channel nodes in the same module device tree file. This property is used for V4L2 camera use cases.

You must set the vc-id property in the gmsl-link node for each sensor to match the vc_id property in the sensor mode device node.

The vc_id property in sensor mode device tree nodes is used for use cases based on the ARGUS camera. For example:

mode0 {                        /* Mode IMX390_MODE_1920X1080_CROP_30 FPS. */
    mclk_khz = "24000";
    num_lanes = "2";
    tegra_sinterface = "serial_a";
    vc_id = "0";
     . . .
}.
VI and CSI Channel Nodes in the Module Device Tree

The module device tree for the reference GMSL setup also performs the port binding to VI and CSI. The number of VI and CSI channels is 2, as in any other dual-sensor setup, but the port-index field would be set based on the hardware connectivity:

vi@15700000 {
    num-channels = <2>;
    ports {
        #address-cells = <1>;
        #size-cells = <0>;
        port@0 {
            reg = <0>;
            imx390_vi_in0: endpoint {
                vc-id = <0>;
                port-index = <0>;
                bus-width = <2>;
                remote-endpoint = <&imx390_csi_out0>;
            };
        };
        port@1 {
            reg = <1>;
            imx390_vi_in1: endpoint {
                vc-id = <1>;
                port-index = <0>;
                bus-width = <2>;
                remote-endpoint = <&imx390_csi_out1>;
            };
        };
    };
};

nvcsi@150c0000 {
    num-channels = <2>;
    #address-cells = <1>;
    #size-cells = <0>;
    channel@0 {
        reg = <0>;
        ports {
            #address-cells = <1>;
            #size-cells = <0>;
            port@0 {
                reg = <0>;
                imx390_csi_in0: endpoint@0 {
                    port-index = <0>;
                    bus-width = <2>;
                remote-endpoint = <&imx390_imx390_out0>;
                };
            };
            port@1 {
                reg = <1>;
                imx390_csi_out0: endpoint@1 {
                remote-endpoint = <&imx390_vi_in0>;
                };                                                          };
        };
    };
    channel@1 {
        reg = <1>;
        ports {
            #address-cells = <1>;
            #size-cells = <0>;
            port@0 {
                reg = <0>;
                imx390_csi_in1: endpoint@2 {
                    port-index = <0>;
                    bus-width = <2>;
                remote-endpoint = <&imx390_imx390_out1>;
                };
            };
            port@1 {
                reg = <1>;
                imx390_csi_out1: endpoint@3 {
                remote-endpoint = <&imx390_vi_in1>;
                };
            };
        };
    };
};

In this configuration the port-index property is set to 0 for both the VI channel 0 and channel 1 nodes, so that both sensors use VI stream 0.

For CSI channel 0 and channel 1 nodes too the port-index property is set to 0, meaning that both channels’ sensors are connected to CSI port A.

The VI and CSI channel nodes also both contain a new property, vc-id, used in V4L2 camera use cases as mentioned above. It must match the gmsl-link device node’s vc-id property.

The bus-width property specifies the number of CSI lanes used.

tegra-camera-platform node configuration in module device tree

The TEGRA-CAMERA-platform device node specifies details of the sensor modules’ configuration. The configuration below is for the reference GMSL module:

tegra-camera-platform {
    compatible = "nvidia, tegra-camera-platform";
    num_csi_lanes = <2>;
    max_lane_speed = <4000000>;
    min_bits_per_pixel = <10>;
    vi_peak_byte_per_pixel = <2>;
    vi_bw_margin_pct = <25>;
    isp_peak_byte_per_pixel = <5>;
    isp_bw_margin_pct = <25>;

    modules {
        module0 {
            badge = "imx390_rear";
            position = "rear";
            orientation = "1";
            drivernode0 {
                pcl_id = "v4l2_sensor";
                /* Driver v4l2 device name */
                devname = "imx390 30-001b";
                proc-device-tree = "/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx390_a@1b";
            };
        };
        module1 {
            badge = "imx390_front";
            position = "front";
            orientation = "1";
            drivernode0 {
                pcl_id = "v4l2_sensor";
                /* Driver v4l2 device name */
                devname = "imx390 30-001c";
                proc-device-tree = "/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx390_b@1c";
            };
        };
    };
};
  • num_csi_lanes is set to 2, since both sensors are connected to a single CSI port A in x2 lane fashion. This is unlike other dual sensors, where num_csi_lanes is set to the total number of CSI lanes used by both sensors, since they use different CSI ports.

  • max_lane_speed is set to an appropriate value for two sensors streaming through a single shared port.

  • The sensor device nodes are defined using sensor proxy I2C slave addresses instead of their physical I2C slave address, and their paths are set in proc-device-tree property. Similar naming is used for the devname property.

For the rest of the configuration, follow the same guidelines as for other sensors.

Refer to the following module device tree files:

  • For Jetson AGX Xavier series: tegra194-camera-imx390-a00.dtsi

Plugin Manager Device Tree

The device tree additions for Plugin Manager are the same as for any other sensor module.

For reference GMSL setup plugin manager support, see the node fragment-imx390@0 in:

  • For Jetson AGX Xavier series: tegra194-camera-plugin-manager.dtsi

Camera Modules Device Tree

All the device nodes in the I2C node of the platform’s camera module device tree are defined in the same way as in the platform device tree. As in the reference GMSL setup, they all go to i2c@0.

The camera module device tree is:

  • For Jetson AGX Xavier series: tegra194-p2822-camera-modules.dtsi

Note

The vc-id property is set to 0 by default for all VI ports in all modules’ device tree files. You must assign correct values to this property in plugin-manager and the sensor-specific module device tree file.

Constraints

Sensor streams connected to same CSI aggregator and sharing a CSI port must use the same lane configuration. The deserializer driver fails to register a stream if its lane configuration does not match other streams that share the same deserializer device.

Apart from that, CSI aggregators generally support multiple identical cameras running with the same frame rate and sensor mode. Review your aggregator data sheet for additional restrictions if you are using a more complicated configuration.

Validation

Dual GMSL sensor streaming (preview/capture) is validated using Argus cameras, sharing CSI port A, and the V4L2 application.

The 2x CSI deserializer/aggregator is validated. The 4x CSI aggregator is not validated, as its hardware module is currently not available.

Known Issues

This section summarizes known issues in the GMSL camera framework.

General Issues

  • Preview flickers on scene change.

  • Sensor tuning and image quality are substandard.

Xavier-Specific

Note

This limitation and rework only apply to older GMSL kits which do not draw on the on-board (DESER) 1.2  V supply.

The 1.2 V power supply needs rework. The MAX9296 deserializer board used in the reference setup does not draw the required on-board 1.2 V power, and the Jetson AGX Xavier Developer Kit carrier board has dropped support for supplying 1.2 V to the camera connector.

The following rework is required on Jetson AGX Xavier for vendor modules that expect 1.2V from the carrier board:

  • Use spare LDO for the SATA controller, which can take maximum 1.21 V. The Vmin requirement from the IMX390 reference sensor is 1.14 V, and the range is 1.14 V to 1.26 V, so this LDO supply is enough to supply 1.2 V to the Jetson AGX Xavier carrier board’s camera connector.

Plugin Manager Board ID

The reference GMSL module currently does not have a unique board ID. Use the MIPI adapter’s board ID, LPRD-001. This ID is used by other sensor modules which use similar MIPI adapters, such as IMX274 and IMX185. To work around this issue until it is resolved, the IMX390 module is disabled in these sensor modules; see tegra186-quill-camera-plugin-manager.dtsi.