IPP Capture (nvmipp_capture)
The NvMedia IPP Capture tool nvmipp_capture can be used to capture raw and processed outputs simultaneously to a file from the NvMedia Image Processing Pipleine (NvMedia IPP). It demonstrates how to use NvMedia IPP to capture raw, ISP-processed and LTM-processed output from a camera with the auto-control algorithms turned on or off. IPP capture can also be used to re-process the captured raw-stream through the NvMedia image processing pipeline using the File Reader component, instead of the Capture component, as input into the IPP pipeline.
The application requires you to provide a command line input with:
• A configuration file
• Capture parameter sets
• The number of cameras to aggregate
The configuration file contains multiple capture parameter sets, one of which is used in the command line. The application parses configuration settings and other command line parameters, and builds the IPP pipeline with the required components. It starts the IPP pipeline to keep capturing frames, writes captured, processed frames to a file if the corresponding save flag has been enabled, and sends them to the EGL stream output for display.
The following diagram provides an overview of the process.
Features
The NvMedia IPP capture tool functional features are as follows:
• Captures raw and processed output(s) to file(s) simultaneously, or one at a time.
• Starts writing frames after auto-algorithm convergence.
• Starts writing frames as available.
• Re-process captured raw stream through IPP.
• Supported file formats:
• Raw output: Bayer raw video stream, nvraw image.
• Processed output: JPEG, TIFF, and YUV420 stream.
Constructing an IPP Capture Pipeline
Each NvMedia IPP Capture pipeline consists of:
• Image Capture Processing (ICP)
• Image Sensor Control (ISC)
• Image Signal Processor (ISP)
• Control algorithm components
• Output components
The EGL producer and GL consumer are attached to IPP pipeline output to provide streams for display. For a detailed description of each IPP component, see
Image Processing Pipeline Framework in this document. The IPP Capture pipeline can be constructed in sensor mode or re-process mode.
IPP Capture in Sensor Mode
In sensor mode, the input to the IPP pipeline is from the sensor. The nvmipp_capture pipeline is configured in sensor mode by default, unless the ‑‑reprocess option has been set. If the save flag is enabled, output components are attached to the ICP and ISP components for saving corresponding output frames to a file.
Raw output capture can be enabled using the save-icp <file-format> flag. Raw capture can be saved as Bayer raw stream or in nvraw image file format by setting the correct file-format parameter. ISP output can be captured to a file by setting the save-isp <file-format> flag through the command line.
Similarly, ltm-output can be captured to a file using the save-ltm <file-format> flag. File formats supported for ISP output and LTM output are:
• JPEG
• TIFF
• Uncompressed YUV420 stream
The following diagram provides the IPP capture pipeline structure in sensor mode.
IPP Capture in Re-Process Mode
In re-process mode, the IPP pipeline does not require a sensor. The input to the pipeline comes from the file instead of the sensor. Re-process mode support Bayer raw stream as the input file format. The IPP pipeline can be configured in re-proess mode by setting the ‑‑reprocess <raw-stream> in the command line.
The following diagram provides the IPP capture pipeline structure in re-process mode.
Features
The NvMedia IPP Capture application supported features include:
• Handles the RAW12 input format
• Provides parameter sets dvp-svc210-raw12-1280x1080-abfor Conti cameras
• Provides parameter sets dvp-ar0231-raw12-1920x1208-ab for AR0231 cameras
These above parameter sets are in <board>.conf.
Where <board>.conf is:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
The nvmipp-capture handles only RAW input in RAW12 input format.
Command Options
The command line options to run the script are as follows.
./x11/nvmipp_capture -cf [filename] -c [name] --aggregate [n] [options]
Required Commands
The following table describes the required command line arguments.
Option | Parameter | Description |
-cf | [filename] | Specifies the configuration file.
Use default.conf at the same folder as nvmipp_capture. |
-c | [name] | Specifies the parameters set name used by capture configuration.
Use the -lps option to get the available configuration parameter sets. |
--aggregate | [n] | Specifies the number of aggregated images. |
Optional Commands
The following table describes optional command line arguments.
Option | Parameter | Description | Default Settings |
-h | N/A | Prints usage and help menu. | N/A |
--ext_sync | [duty_ratio] | Enables external synchronization. (AR0231 or OV10635 sensors only.) | 0.25 |
-v | [level] | Specifies the logging level of: 0—Errors 1—Warnings 2—Information 3—Debug | If no level is provided, defaults to 3. If option is not provided, defaults to 0. |
-lps | N/A | Lists the available configuration parameter sets. | N/A |
-d | [id] | Enables the display. If provided, uses the display with the specified ID. | Default: Display is disabled.
If ID is not provided, the sample uses the first display available. |
-f | [file-prefix] | Specifies the filename prefix for images captured from the pipeline. | Default: Output |
-n | [frames] | Specifies the number of frames to capture. | Default: 10 |
-s | [n] | Sets the number of frames to skip before capturing images. | Default: 10 |
-w | [n] | Set display window ID [0-2]. | Default: 0 |
-z | [n] | Set display window depth [0-255]. | Default: 0 |
--cam_enable | [n] | Enable or disable camera[3210]; enable:1 disable:0 | Default: n = 0001 |
--cam_mask | [n] | Mask or unmask camera[3210]; mask:1 unmask:0 | Default: n = 0000 |
--csi_outmap | [n] | Set csi out order for camera[3210]; 0,1,2 or 3 | Default: n = 3210 |
--sample-plugin | | Use IPP sample plugins for control algorithm components | N/A |
--version | | Show library modules version information | N/A |
--save-icp | <file-format> | Saves the raw-output frames from the pipeline. Supported file formats include: nvraw and raw. nvraw—saves raw-output frames in nvraw image file format as follows: <file-prefix>_<cap-module>
_<pipeline>_<frameNo>.nvraw raw—saves raw-output as Bayer raw video stream as follows: <file-prefix>_<cap-module>
_<resolution>_<bayerOrder>
_aggr<n>.raw | Default: nvraw |
--save-isp | <file-format> | Saves the ISP output frame from the pipeline.
Supported file formats include: JPG, TIFF, YUV. jpg—saves ISP output frames in JPEG file format as: <file-prefix>_<cap-module> isp<pipeline>_<frameNo>.jpg tiff—saves ISP output frames in TIFF file format as: <file-prefix>_<cap-module> isp<pipeline>_<franeNo>.tiff yuv—saves ISP output frames in Bayer yuv420 video stream as: <file-prefix>_<cap-module> isp<pipeline>_<resolution> _<yuvformat>.yuv | Default: jpg |
--save-ltm | <file-format> | Saves the LTM output frames from the pipeline.
Supported file formats include: JPG, TIFF, YUV. jpg—saves LTM output frames in JPEG file format as: <file-prefix>_<cap-module> _ltm_<pipeline>_<frameNo>.jpg tiff—saves LTM output frames in TIFF file format as <file-prefix>_<cap-module> _ltm_<pipeline>_<frameNo>.tiff yuv—saves LTM output frames as Bayer yuv420 video stream as <file-prefix>_<cap-module> _ltm_<pipeline>_<resolution> _<yuvformat>.yuv | Default: jpg |
--reprocess | <raw-stream> | Specifies the re-process Bayer raw video stream through the IPP pipeline. For aggregate raw stream, the --aggregate [n] flag is required. | N/A |
--mailbox | N/A | Specifies to manage frames in Mailbox mode mode. | |
--show-timestamp | N/A | Shows the timestamp information. Use with ‑v option. | N/A |
--show-metadata | N/A | Shows the metadata information. Use with ‑v option. | N/A |
--nvbracketae | N/A | Uses NvMedia IPP bracketed auto-exposure plugin for control algorithm components. | N/A |
Configuration Parameters
Each configuration file is required to start with a list of configuration parameters. The application uses these parameters to determine the capture settings.
The required parameters are:
; Interface: <type>
; Input Format: <format>
; Resolution: <resolution>
; CSI Lanes: <number of lanes>
; I2C Device: <device number>
; Sensor Address: <address>
• Interface—specifies the CSI interface type used.
• Input Format—specifies the format the sensor is capturing images.
• Resolution—specifies the resolution the sensor is capturing. This includes embedded lines.
• CSI Lanes—specifies the number of CSI lanes used.
• I2C Device—specifies the ID of the I2C device that is connected to the aggregator.
• Sensor address—specifies the address of the sensor used for I2C.
You can also provide optional configuration parameters:
; Surface Format: <surface format>
; Max9286 Address: <address>
• Surface format—specifies the CSI surface format. If this is not provided, the application chooses a default surface format based on the input format.
• Max9286 address—provides additional debug information by dumping the registers of the aggregator.
The following is an example configuration for the p2360 board:
; Interface: csi-ab
; Input Format: raw12
; Resolution: 1280x1084
; CSI Lanes: 4
; I2C Device: 2
; Sensor Address: 0x60
; Max9286 Address: 0x90
Example Commands
In these examples, where <board>.conf is identified, replace with one of the following:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
The command below uses CSI AB interface connected with four OV10640 cameras, each of which outputs 1280x1080 RAW 12 data.
./x11/nvmipp_capture -cf <board>.conf -c dvp-c1-ov10640-b1-raw12-1280x1080-ab --aggregate 4
The command below uses CSI CD interface connected with one AR0231 cameras which outputs 1920x1208 RAW 12 data:
./x11/nvmipp_capture -cf <board>.conf -c dvp-ref-ar0231-raw12-1920x1208-cd -d 0
Saving Raw Output
In this example, where <board>.conf is identified, replace with one of the following:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
To save raw output, add --save-icp <format> to the basic command line parameters. For example, skip the initial 10 frames (-s 10) and then capture 200 raw output frames (-n 200) as raw-output stream to file with the prefix test2 under the directory /media/usb0.
./x11nvmipp_capture -cf <board>.conf -c dvp-svc210-raw12-1280x1080-ab -d 0 --save-icp raw -s 10 -n 200 -f /media/usb0/test2
Saving Raw and ISP Output Simultaneously
In this example, where <board>.conf is identified, replace with one of the following:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
To save ICP output, add --save-icp <format> and --save-isp <format> to the basic command line parameters. For example, skip the initial 10 frames (‑s 10) and then capture 100 raw (nvraw) and corresponding isp-output (jpg) frames with the prefix "test3" under the directory /media/usb0.
./x11/nvmipp_capture -cf <board>.conf -c dvp-svc210-raw12-1280x1080-ab -d 0 --save-icp nvraw --save-isp jpg -s 10 -n 100 -f /media/usb0/test3
Reprocess Bayer Raw Stream Through IPP Pipeline
In this example, where <board>.conf is identified, replace with one of the following:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
To save as reprocess, add --reprocess <raw-stream> to the basic command line parameters. For example, reprocess Bayer raw stream test_c_max9286_9271_ov10640_1280x1084_BGGR.raw captured using the capture configuration dvp-svc210-raw12-1280x1080-ab.
./x11/nvmipp_capture -cf <board>.conf -c dvp-svc210-raw12-1280x1080-ab -d 0 --reprocess test_c_max9286_9271_ov10640_1280x1084_BGGR.raw
Troubleshooting
Users of this application have encountered the following issues.
Video Link Errors
In this error, where <board>.conf is identified, replace with one of the following:
• For AutoCruise (P3407): drive-px2-autocruise.conf
• For AutoChauffeur (P2379) Tegra A: drive-px2-a.conf
• For AutoChauffeur (P2379) Tegra B: drive-px2-b.conf
Several situations can cause generic video link errors.
The error message is as follows:
./x11/nvmipp_capture -cf <board>.conf -c dvp-svc210-raw12-1280x1080-ab --aggregate 1
nvmedia: ERROR: ConfigISC_m: Video Link(0) is not detected
nvmedia: ERROR: IPP_Init: Failed to create ISC Device
nvmedia: ERROR: IPP_Init: Failed
Possible causes include:
• One or more cameras are not connected to the right CSI port or not started at port numbered 0. The number of cameras connected must match the number specified in the --aggregate option.
• The type of sensor connected and the configuration parameter specified in the command line conflict.
• One of the cameras failed to start or is not functioning.
