frontend (TensorRT multichannel video capture)

• Overview
• Building and Running
• Flow
• Key Classes and Methods

Requirements

• Camera
• Display

---

Overview

This sample demonstrates video capture for four video streams coming directly from camera sensors, where the streams have different resolutions. Three of the streams go through the encoding process and are saved as H.264 or H.265 files. The fourth stream provides full resolution and uses NVIDIA^® TensorRT^™ to perform object detection. If target object is detected, it will draw a bounding box around the object on the frame. Finally, the frame with the bounding box is rendered on the display and simultaneously encoded as an H.264 or H.265 file.

---

Building and Running

Prerequisites

• You have followed Steps 1-3 in Building and Running.

To install TensorRT

1. Open the apt source configuration file in a text editor:

     $ sudo vi /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
   

2. Change the repository name and download URL in the deb commands shown below:

     deb https://repo.download.nvidia.com/jetson/common <release> main
     deb https://repo.download.nvidia.com/jetson/<platform> <release> main
   

   <release>> is the release number. Ex: r32.5.

   <platform> identifies the platform's processor.

   • t194 for Jetson AGX Xavier series or Jetson Xavier NX
   • t186 for Jetson TX2 series
   • t210 for Jetson Nano or Jetson TX1
3. Enter:

     $ sudo apt-get update
     $ sudo apt-get install tensorrt
   

To build

1. Ensure the following variable is set to 1 by making the following change in the Makefile:

     ENABLE_TRT := 1
   

2. Enter:

   $ cd frontend
   $ make
   

To run

• Enter:

   $ ./frontend --deploy ../../data/Model/GoogleNet_three_class/GoogleNet_modified_threeClass_VGA.prototxt \
       --model ../../data/Model/GoogleNet_three_class/GoogleNet_modified_threeClass_VGA.caffemodel \
       --no-preview
  

To quit

• Enter q.

To view command-line options:

• Enter:

   $ cd frontend
   $ ./frontend -h
  

---

Flow

The following diagrams illustrate the flow of data through this sample.

Detailed Buffer Management

Buffer Flow of the Frontend Using TensorRT

   $ ./frontend --deploy ../../data/Model/GoogleNet_three_class/GoogleNet_modified_threeClass_VGA.prototxt \
       --model ../../data/Model/GoogleNet_three_class/GoogleNet_modified_threeClass_VGA.caffemodel

---

Key Classes and Methods

For information on libargus, see Libargus Camera API. For information on EGLStream, see documentation on the Khronos web site.

StreamConsumer Class

The sample defines StreamConsumer as an abstracted class. It is not imported from other libraries.

Class	Description
StreamConsumer	A base class of the libargus OutputStream consumer. It contains a dedicated thread to poll frames from Argus::OutputStream.

The following table describes key StreamConsumer members.

Members	Description
setOutputStream	Sets OutputStream which be used to create the FrameConsumer.
getSize	Gets stream resolution.
threadInitialize	Virtual method that initializes the consumer thread.
threadExecute	Virtual method that manages the thread loop.
threadShutdown	Virtual method that destroys the consumer thread.
processFrame	Virtual method that handles one frame.

VideoEncoder Class

The sample defines VideoEncoder as an abstracted class. It is not imported from other libraries.

Class	Description
VideoEncoder	This is a utility class to simplify the use of V4L2 video encoder.

The following table describes key VideoEncoder members.

Members	Description
VideoEncoder	Constructor in which to specify resolution, output video filename, and select encoding format.
initialize	Initialize video encoder and stream consumer.
shutdown	Destroy resources.
setBufferDoneCallback	Set callback to return buffer to caller.
encodeFromFd	Feed the encoder with a new buffer.

VideoStreamConsumer Class

The sample defines VideoStreamConsumer as an abstracted class. It is not imported from other libraries.

Class	Description
VideoStreamConsumer	This class is a derived class of StreamConsumer to support video encoding. It reuses the VideoEncoder object for video encoding.

The following table describes key VideoStreamConsumer members.

Members	Description
VideoEncodeStreamConsumer	Constructor in which to specify the name of the consumer, output video filename and select encoding format.
threadInitialize	Initializes the video encoder and stream consumer.
threadShutdown	Destroys resources.
processFrame	Receives frames and pushes them to the encoder.

TRTStreamConsumer Class

The sample defines TRTStreamConsumer as an abstracted class. It is not imported from other libraries.

Class	Description
TRTStreamConsumer	This class is a derived class of StreamConsumer to support TensorRT (GIE) inference. It also has a VideoEncoder object to support video encoding.

The following table describes key TRTStreamConsumer members.

Members	Description
TRTStreamConsumer	Constructor in which to specify the name of the consumer, output video filename and select encoding format.
threadInitialize	Initializes TensorRT context, video encoder and stream consumer.
threadShutdown	Destroys resources.
processFrame	Receives frames and prepares them for the TensorRT inference.
setDeployFile	Sets the deploy file for building the TensorRT context.
setModelFile	Sets the model file for building the TensorRT context.
RenderThreadProc	Loops to draw the bounding box, render the frame, and encode the video.
TRTThreadProc	Loops to do the TensorRT inference.