1 # Copyright (c) 2019-2020, NVIDIA CORPORATION. All rights reserved.
3 @page camera_mainsection Camera
6 @note SW Release Applicability: This module is available in both **NVIDIA DriveWorks** and **NVIDIA DRIVE Software** releases.
8 @section camera_mainsection_about About This Module
10 The Camera module defines a common interface that expands the categories of
11 cameras that NVIDIA<sup>®</sup> DriveWorks supports.
13 The Camera module supports both GMSL and USB interfaces. It also provides the ability to replay recorded camera data through the DriveWorks stack as a virtual sensor.
15 A selection of GMSL cameras are supported out-of-the-box; please refer to @ref camera_supported_output_types for a list of these sensors. DriveWorks also provides the ability to integrate non-natively supported GMSL cameras. Please refer to @ref sensorplugins_camerasipl for more information.
17 @section camera_mainsection_camera_creation Creating and Using Camera
19 Specify the following when creating a camera using the Sensor Abstraction Layer (SAL):
20 - Camera protocol, which can be `camera.gmsl`, `camera.virtual`, or `camera.usb`.
21 - A set of parameters specific for each camera protocol.
23 All camera protocols have a common API, created via the SAL through a parameter string, which changes based on the protocol.
24 Once all the camera objects have been successfully created, and the application's initialization phase is complete, start the sensors with `dwSensor_start()`.
26 @note Starting a sensor triggers asynchronous capture or prefetching, so it should be performed right before deciding to call `dwSensorCamera_readFrame()`. Camera sensors do not start right away, so check for `DW_NOT_READY` in a loop until
27 the first instance of `DW_SUCCESS` or any failure.
29 `dwSensorCamera_readFrame()` provides a `dwCameraFrameHandle_t()` which is an opaque structure mapping to a Frame. A Frame is a container of images and support functions, unique to the protocol at use. The purpose of a Frame is to acquire a camera event and be able to retrieve an image suitable for the use case at hand. A call to such a function is always followed by a necessary timeout value in microseconds, due to the
30 asynchronous nature of some of the protocols, to ensure no deadlocking happening.
32 After a frame is grabbed to the application level, applications use
33 `dwSensorCamera_getImage()` to get an image in the specified output type. The available images that can be returned by this operation depend on the setup and platform. DriveWorks allows three levels of outputs:
34 - Native level: corresponds to the image that comes out of the lower level with as little extra operations done as possible. This is the most performant layer for memory and speed.
35 - Streamed level: corresponds to images who have been streamed to CUDA in order to be unified in dwImageType, guaranteeing a common output in case of differences in setup or protocol.
36 - RGBA level: a conveniently chosen format which is the result of streaming to CUDA and converting from the native format which is widely used.
38 For a detailed table on supported output types, please refer to @ref camera_supported_output_types.
40 @subsection camera_mainsection_gmsl_cameras GMSL Cameras (camera.gmsl)
42 The `camera.gmsl` protocol describes GMSL cameras. These cameras acquire frames at a frequency based on the camera's frame rate
43 For more information on image processing and management, please refer to "Understanding NvMedia" in the latest _NVIDIA DRIVE OS SDK Development Guide_.
45 This protocol is based on top of the NvSIPL library, which takes care of the driver loading and low level handling of the camera.
46 NvSIPL connects the following components:
47 - Image Sensor Control (ISC).
48 - Image Sensor Processing (ISP).
49 - Control Algorithm that regulates exposure and white balance based on sensor
50 statistics coming from Sensor Control and ISP.
52 The parameters for `camera.gmsl` are:
54 camera-name = [name as specified in the SIPL database]
55 interface = [csi-a - csi-h | csi-ab - csi-gh | trio-a - trio-h | etc...]
57 output-format = [processed|raw]
58 isp-mode = [see camera.gmsl]
59 nito-file = [optional path to .nito file]
61 CPHY-mode = [0: default DPHY Mode / 1: CPHY]
62 deserializer = [optional name of a deserializer to be explicitly selected]
63 encoder-instance=<0: default or 1>
65 Each `dwSensorHandle_t` corresponds to a camera described above by name, port and link. The choice for `csi-port` depend on the sensor, especially if it is a custom one. Refer to the NvSIPL guide for more info.
67 All cameras are attached to and controlled by a master controller. If one camera fails, it results in a failure for the whole system.
69 Once a sensor is started or stopped, it sends the corresponding signal to the master controller, which starts or stops all cameras simultaneously. Ensure that you always call `dwSensor_start()` and `dwSensor_stop()` for all sensors.
71 The `camera-name` parameter enables DriveWorks to tell NvSIPL which camera to use, and then load the camera drivers (under /usr/lib/nvsipl_drv).
72 Once the drivers are loaded, the camera setup is read and communicated to DriveWorks, which then allocates the necessary resources needed by NvSIPL. Finally, a raw image coming from the sensor's ISC is fed to the ISP layer at runtime, converting it into a processed image.
74 @note In general, DriveWorks modules require processed images to function properly.
76 DriveWorks allocates image buffers that are registered with the NvSIPL library. These are in equal amounts buffers for the raw images and buffers for the processed images. You can control the number via the `fifo-size` parameter.
77 Once the camera starts, NvSIPL grabs the images from the buffers and puts them to use, continuing at a rate based on the camera frame rate (in addition to some slight processing delay based on hardware specifications).
79 The asynchronous mechanism continues until the buffers run out, where all camera captures will not have an image in the buffer available for use. They are then dropped as a result, otherwise known as an ICP drop.
81 NvSIPL fills the raw image buffers and processed image buffers, and sends them back to DriveWorks. DriveWorks then stores them in a queue, in the order of their capture time.
83 You also have control over the `dwSensorCamera_readFrame()`, which temporarily retrieves the front of the frame queue for you. Ensure you eventually return it back to the sensor, in order to allow NvSIPL to have buffers to work with again. Generally, the buffer usage rate must be lower than the camera frame rate in order to ensure no frame is ever dropped.
85 The user can read as many frames as the `fifo-size` together and then returning them, as long as NvSIPL is not starved. A basic locking mechanism ensures that if reading is done in a multi-threaded fashion, no race condition should happen.
87 As mentioned in the table above, the image outputs available depend on the `output-format` which, by default, is `processed+raw`. Choosing only `processed` or `raw` reduces the number of resources allocated by a single camera. If you do not require raw images to be retrieved in the application level, then choosing `raw` in `output-format` is wasteful.
89 When the processed pipeline is set, NvSIPL allocates the resources needed to process the camera selected. This includes the NITO file (Nvmedia Isp Tuning Object), which contains code to correctly process a raw image. NITO files stored in the DDPX under `/opt/nvmedia/nit`, and the file corresponding to a certain camera is be computed based on the camera name.
91 For example, `camera-name=SF3324` will search for
92 `/opt/nvmedia/nit/SF3324.nito` and `/opt/nvmedia/nit/sf3324.nito`. If none of these are detected, DriveWorks attempts to use a generic NITO file: `/opt/nvmedia/nit/default.nito`. In most cases however, this NITO is not suitable and will likely cause an application crash. If this occurs, ensure you provide the right camera name or the path to its own NITO file. If you do not have a NITO
93 file (e.g., working with a custom camera), you can still retrieve raw images (`output-format=raw`) but you will not be able to use DriveWorks to process them, will require a custom code to do so.
95 The `dwSensorSerializer` API allows you to record data. Depending on the `output-format` selected, the `camera.gmsl` allocates resources to record raw sensor feeds, processed video files, or both.
96 A raw sensor feed is the unprocessed signal coming from the camera itself, and a processed video is an h264/h265/mp4 video of the processed output of the camera. A processed output is unchangeable and does not carry information about the sensor it was recorded with, whereas a raw signal feed can be reprocessed again, just like a live camera feed. It contains all information regarding the sensor, including its embedded metadata.
98 @subsubsection gmsl_camera_logging Relevant logging
100 Logging is crucial when working with NvSIPL, providing insights into whether things are working as intended.
101 When initializing, the log should print:
103 devBlock: 1 Slave = 0 Interface = csi-a Camera_name = SF3324 Link = 0
105 This helps double check if the sensor is going to be searched correctly. A devblock corresponds to a camera port, with a total maximum of 4 blocks. If the camera is found in the database:
107 Camera Match Name: SF3324 Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705 linkIndex: 4294967295 serInfo.Name: MAX96705
109 This means the driver search can be done. Ensure you should double check if this information is correct. Afterwards, when all cameras are detected, the master controller starts and prints the complete information for each camera detected if the driver was correctly loaded:
111 CameraGSMLMaster: starting...
118 Deserializer Name: MAX96712
119 Deserializer Description: Maxim 96712 Aggregator
120 Deserializer i2cAddress: 41
124 Number of camera modules: 1
128 These are confirmation prints, followed by prints coming directly from NvSIPL and the camera driver:
130 MAX96712: Revision 2 detected
131 MAX96712: Enable periodic AEQ on Link 0
132 MAX96705: Pre-emphasis set to 0xaa
133 MAX96705: Revision 1 detected!
134 Sensor AR0231 RCCB Rev7 detected!
136 These are completely different depending on the camera at use and the DRIVE OS version, but a good indicator that things are going well in general. Then the NITO file is searched:
138 CameraClient: no nito found at /opt/nvidia/nvmedia/nit/SF3324.nito
139 CameraClient: using nito found at /opt/nvidia/nvmedia/nit/sf3324.nito
141 If the NITO file is found, things can progress as explained in the sections before. Then, when `dwSensor_start()` is called, if things go correctly:
143 CameraMaster: bootstrap complete
145 At this point, each camera that starts acquiring frames will print:
147 CameraClient: Acquisition started
149 There are higher levels of verbosity that can be adjusted by exporting DW_SIPL_VERBOSITY=[0..4].
151 @subsection camera_mainsection_virtual Virtual Cameras (camera.virtual)
153 The `camera.virtual` protocol handles recorded data in raw or processed format. The user specifies `video=` or `file=` with the location of the data. It is possible to specify multiple `video=` or `file=` parameters, e.g. to reference a raw and a processed video. The SAL will prefer `video` over `file` parameters and use the first valid filepath, i.e. the referenced file needs to exist or it is skipped. This may be useful in cases where some video files are not always available, e.g. because one video format is provided via mounted network paths which may be slow or unreliable.
157 A h264/h265/mp4 video is a video file that is decoded by `camera.virtual` into processed frames, identical to how they were when the sensor was being processed. This data is unchangeable and can be completely artificial data. Potentially, DriveWorks can replay any h264/h265/mp4 data.
159 Below is the parameter string format for when working with processed videos:
161 params: video = <path.h264/h265/mp4>
162 output-format = [processed only]
164 ### Raw Signal Feeds (AKA Raw videos)
166 A raw video contains the same signals coming from the camera at the moment of capture. This means that the data is not interpretable, unless the same resources used to process the actual physical data are present at the time. In this case, the NvSIPL library uses the camera drivers to treat the frame blobs in the raw feed, as if they are sensors coming from that moment from a live camera.
168 This process is indistinguishable from a live camera; the logs, errors, sensor events, and asynchronous behavior all match the live case.
169 For this reason, you can specify the extra parameters available in `camera.gmsl`: namely `isp-mode`, `fifo-size`, and `nito-file`.
170 DriveWorks reads the file header to recognize the camera and its information. The name for the camera used to recover the driver is stored as a string in the header.
171 However, you can override this by passing in the `camera-name` parameter. It is similar to the live case, where the camera requires a NITO file, and the process is the same as described above.
173 Below is the parameter string format for when working with Raw videos:
175 params: video = <path.raw/lraw>,output-format=[processed|raw],isp-mode=<see camera.gmsl>,nito-file=<path.nito>, fifo-size=[size],camera-name=[overridden camera name]
177 @subsection camera_mainsection_gmsl_usb_ptgrey USB Cameras (camera.usb)
179 The `camera.usb` protocol describes USB cameras. These cameras have similar
180 interfaces. When a sensor is started, the acquisition is performed by the OpenCV
181 (USB) library. This library uses the OS drivers and stores the
182 result in a buffer allocated in Camera. The driver automatically detects the
183 frame rate for these USB cameras.
185 @section camera_mainsection_gmsl_use_cases Relevant Tutorials
187 - @ref camera_usecase1
188 - @ref camera_usecase2
189 - @ref camera_usecase3
190 - @ref camera_usecase4