DeepStream

Overview

Metropolis Microservices provide a DeepStream application for real-time processing of various camera streams presented by VST based on DeepStream release 6.4. The processing includes object detection using the PeopleNet model from NVIDIA, followed by object tracking using the NvDCF tracker plugin available in DeepStream. The output of the application is metadata based on metropolis schema sent to Redis message bus using msgbroker plugin.

A customized DeepStream app known an deepstream-moj-app is provided, which is an extension of the deepstream-test5 reference application, and is able to handle multiple pipelines simultaneously. Each pipeline is specified using the -c option. As part of the config provided, the app uses two pipelines corresponding to inference executing on each of the two DLAs on Orin AGX or NX16. In addition to the standard options supported by deeptream-test5, the additional options added are:

-g option specifies to enable gem overlays (see below)
-e option specifies the IP and port for the Analytics Web API endpoint to be able to fetch gem configs and current counts

-s option specifies the IP and port for the VST API to be able to fetch current streams at runtime.

# -c <PATH/TO/PIPELINE/CONFIG/FILE>
#example for running 2 pipelines on Orin AGX
$ deepstream-moj-app -g -e localhost:5000 -s localhost:81 -c /ds-config-files/pn26/ds-config-0_agx.txt -c /ds-config-files/pn26/ds-config-1_agx.txt

These pipelines use DLA 0 & 1 through their gie config files:

enable-dla=1 # 0 => disable, 1 => enable
use-dla-core=0 # dla core number => 0, 1, etc.

There are four ds-config- config files in the configs/deepstream/pn26 folder of the Docker Compose repo. Two of these are marked as agx and two as nx specifying the device type they should be used on. This folder is mounted in the DeepStream Docker container. Depending on which compose file run (compose_agx.yaml for Orin AGX and compose_nx.yaml for Orin NX16), the corresponding DeepStream config files are passed into the DeepStream application.

The deepstream-moj-app uses the DeepStream test5 reference application as a starting point and implements gem overlay as an addition. An additional container called SDR is used to automatically add and remove streams from the DeepStream application based on events that occur in VST. An overview of how DeepStream and VST interface with other components can be seen below:

Configuration

No changes should need to be made to the DeepStream config files. Up to 8 streams per DLA are supported (16 total) on Orin AGX and 3 streams per DLA are supported (6 total) on Orin NX16. Notable points about the configuration include:

use of DLA to offload inference from GPU
Use of PeopleNet 2.6 unpruned model for superior accuracy
Inference done every alternate frame based on the interval parameter within the primary-gie section
Use of NvDCF multi-object tracker that supports running on PVA backend for further reducing the load on GPU
Use of Redis message broker for metadata output
Use of dynamic stream addition to add streams using the SDR microservices (see below)

In order to increase or reduce the number of streams per pipeline from the default values, the WDM_WL_THRESHOLD value under compose_agx.yaml or compose_nx.yaml will need to be changed. Similarly, the max_devices_supported value in the VST config (config/vst/vst_config.json) may also need to be edited to support the necessary number of streams.

SDR (Sensor Distribution and Routing)

Stream addition to DeepStream is based on a dedicated microservice named SDR to discover streams from VST and add them to DeepStream on the fly. SDR is used to automatically add and remove streams from DeepStream using the dynamic stream addition feature in DeepStream, triggered based on events from VST. When a stream is added to VST and is able to stream, an event is sent by VST to Redis with the following format:

{
    "alert_type": "camera_status_change",
    "created_at": "2023-11-20T23:51:04Z",
    "event": {
        "camera_id": "MOJD7",
        "camera_name": "36398191-d48a-4260-b574-239fd59f156f",
        "camera_url": "rtsp://172.17.170.143/live/36398191-d48a-4260-b574-239fd59f156f",
        "change": "camera_streaming"
    },
    "source": "vst"
}

SDR picks this up, then determines which of the two pipelines in DeepStream currently has availability and adds it to that.

Sometimes when SDR starts up, VST may already have streams added and will not resend a message on Redis. This can occur for instance if SDR was restarted. To solve this, on startup SDR queries the VST live streams endpoint to fetch streams that have already been added. It then adds these streams as well to DeepStream.

SDR keeps track of which pipeline each stream was added to. This is so that when a stream is deleted, it knows which pipeline to send the delete request to. Each DeepStream pipeline has a different port that needs to be specified when using the add or remove API call.

To add a stream to DeepStream, a POST request must be made to the http://localhost:9010/api/v1/stream/add endpoint. Note that localhost may need to be changed to the proper IP and the port from 9010 to something else - by default port 9010 is used for pipeline 1 and port 9011 for pipeline 2, this is specified in the DeepStream config files. The following JSON must also be included in the request:

{
    "key": "sensor",
    "value": {
        "camera_id": "Amcrest_1",
        "camera_name": "Amcrest_1",
        "camera_url": "rtsp://10.136.27.7/live/7995217a-3f05-47a5-885d-50c937bf2ed5",
        "change": "camera_add",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}

To remove a stream from DeepStream, a POST request must be made to the http://localhost:9010/api/v1/stream/remove endpoint. Again, localhost may need to be changed to the proper IP and specify the correct port. The following JSON must also be included in the request:

{
    "key": "sensor",
    "value": {
        "camera_id": "Amcrest_1",
        "camera_name": "Amcrest_1",
        "camera_url": "rtsp://10.136.27.7/live/7995217a-3f05-47a5-885d-50c937bf2ed5",
        "change": "camera_remove",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}

Note that only the camera_id, camera_name, camera_url, and change values are necessary, everything else is optional. Also note that these two API endpoints are interfacing directly with DeepStream and thus if used manually, will skip over SDR and may cause issues. Internally, SDR uses these endpoints and JSON formats to interface with DeepStream. These add and remove endpoints are provided in the default DeepStream test5 application when using [source-list] in the config file.

SDR also supports recovery from crashes. In the event that SDR, DeepStream, or the VST containers crash or restart, streams should automatically be removed from DeepStream then re-added such that the stream links are up to date with VST and locations for which pipeline each stream is processed on in DeepStream is also synced up with SDR.

There are various SDR configuration options that can be set via container environment variables and can be seen used in the Docker Compose yaml files. These options are specified below:

WDM_WL_SPEC option specifies the local cache file SDR uses in case of a crash/restart.
WDM_CLUSTER_CONFIG_FILE option specifies IP/port values for each DeepStream pipeline.
WDM_MSG_KEY option specifies the Redis key SDR should listen on.
WDM_WL_REDIS_MSG_FIELD option specifies the key within the Redis JSON to parse.
WDM_WL_ADD_URL option specifies the DeepStream add endpoint.
WDM_WL_DELETE_URL option specifies the DeepStream remove endpoint.
WDM_WL_HEALTH_CHECK_URL option specifies the DeepStream health check endpoint.
WDM_WL_CHANGE_ID_ADD option specifies the “change” type to listen for on Redis to know when to add a stream to DeepStream.
WDM_PRELOAD_WORKLOAD option specifies a config file with streams which should be auto loaded on startup other than those already available on VST.
WDM_CLEAR_DATA_WL option specifies if the local cache should be cleared at startup.
WDM_KFK_ENABLE option specifies if Kafka should be enabled.
WDM_DS_SWAP_ID_NAME option specifies if ID and Name should be swapped from the VST Redis message before adding to DeepStream.
WDM_VALIDATE_BEFORE_ADD option specifies if the add JSON should be validated before being sent to DeepStream (ie., make sure name, ID, and URL are set).
WDM_PRELOAD_DELAY_FOR_DS_API option specifies if SDR should wait for the DeepStream health check to pass before starting to add streams.
WDM_WL_THRESHOLD option specifies the maximum number of streams SDR should add per pipeline.
WDM_CLUSTER_TYPE option specifies the cluster type being used.
WDM_WL_OBJECT_NAME option specifies the workload object name for SDR events sent to Redis and must be unique across instances of SDR running on the same system.
WDM_CONSUMER_GRP_ID option specifies the Redis consumer group ID and must be unique across instances of SDR running on the same system.
WDM_CLUSTER_CONTAINER_NAMES option specifies the container names to watch for restarts/crashes for the SDR recovery mechanism.

the docker_cluster_config.json file pointed to by WDM_CLUSTER_CONFIG_FILE must be in the following format:

{
    "moj-ds-01": {
        "provisioning_address": "localhost:9010",
        "process_type": "docker"
    },
    "moj-ds-02": {
        "provisioning_address": "localhost:9011",
        "process_type": "docker"
    }
}

Where:

The number of keys are equal to the total number of services you want to distribute to.
The keys for all must be unique.
provisioning_address must be set to the IP/port of each service (ie. in the case of DeepStream, the IP is localhost but the port differs for the two pipelines).
process_type must be set to docker if using Docker Compose.

SDR will fill up each in order. In other words, SDR will first add WDM_WL_THRESHOLD sources to “moj-ds-01” before adding sources to “moj-ds-02”. In the event a source is removed from “moj-ds-01”, that spot will be filled by the next source added.

PeopleNet based Inference

The DeepStream module uses NVIDIA PeopleNet version 2.6 for object detection model. Pre-generated engine files are used to reduce startup time. Engine files vary by device used (Orin AGX, Orin NX16) as does batch size, hence the varying DeepStream configs depending on device. As mentioned before, inference runs on the DLA engines. Tracking distance of 1 is used, which means inference is run every alternate frames to limit inference utilization. Hybrid clustering is used to process network output.

Note that PeopleNet 2.6 engine file used with DeepStream to be run on DLA with TensorRT in Jetpack 6.0 need to generated offline due to a known issue documented in the TensorRT release notes, captured here for convenience:

When building TensorRT engines for DLA, there is a known issue that entire DLA subgraphs listed in “Layers Running on DLA” (seen with TensorRT’s verbose mode) cannot be built/eventually fall back to GPU with the message “{ForeignNode[…]} cannot be compiled by DLA, falling back to GPU”. This has been observed with the two ResNet-based models PeopleNet v2.6 and TrafficCamNet from TAO. In both cases, this issue can be fixed by changing TensorRT’s default DLA SRAM pool size of 1 MiB to 0.5 MiB. Using trtexec, this can be achieved by adding the argument --memPoolSize=dlaSRAM:0.5 when building the TensorRT engine.

NvDCF based tracker

The NvDCF tracker used in DeepStream has been configured for inference being run every alternate frame. NvDCF is particularly suitable for people analytics in indoor, retail settings based on its visual tracking capability that makes it resilient to full or partial occlusions. Refer to the ll-config-file variable definition in the DeepStream config file for tracker configuration being used.

Gem Overlay

The deeepstream-moj-app adds support for gem overlays. As specified above, to enable this the additional -g option needs to be passed in. When this is passed in, the -e option along with IP and port for the Metropolis WebAPI microservice and the -s option along with the IP and port for the VST API must be specified.

Every second, DeepStream queries Metropolis WebAPI microservices to check for updates to each streams gem configs. This includes both Region of Interest (ROI) and tripwire (TW) configs. Each input stream can display a maximum of one ROI and one TW. If multiple exist for a given stream, only the first of each type is chosen.

ROI and TW counts are fetched every 100ms from Metropolis WebAPI microservice. These counts are then displayed in the top left corner of the overlay stream. ROI counts are real time and display the current number of people in the ROI. TW counts are cumulative counts and reset to 0 at the top of the hour.

Overlays for both text and gem lines/regions are added via a probe to the tiler.

Streams are automatically added here via SDR once added to VST. Note that stream names must all be unique in VST otherwise issues may occur in SDR and DeepStream since stream name is used as key values internally. Even if the -g option is not passed in, bounding box overlay is still supported. Similarly, in the event -g is passed in but a given stream has no associated gems, only bounding box overlay will be shown.

Overlay streams can be found at rtsp://<JETSON-DEVICE-IP>:8555/ds-test and rtsp://<JETSON-DEVICE-IP>:8556/ds-test depending on the pipeline the input stream was placed in. The number of input streams for each output stream varies based on what WDM_WL_THRESHOLD is set to under SDR in the Docker Compose config. Note that the ports specified are the default options and can be changed in the DeepStream config.

These overlay streams can be viewed via VLC. Alternatively, they can be added into VST as a new stream to be able to view via the VST web UI. If adding to VST, ensure that the name includes the word “overlay”, otherwise SDR will not ignore this stream and add it back to DeepStream causing a circular dependency.

Logs

DeepStream logs are saved in the /data/logging-volume/deepstream.log file by default. Live logs can be viewed using the command tail -f /data/logging-volume/deepstream.log | grep -ve “^$”. The current streams being processed by DeepStream can be viewed in this file as well as the current fps values for each stream. Generally, these should be around 30fps with default config options, but may be lower if the input stream is a lower fps value. A sample snippet of the DeepStream logs can be seen below:

Active sources : 4
Mon Feb 12 18:18:32 2024
**PERF:
30.09 (32.54)   source_id : 3 stream_name MOJD3
30.09 (32.81)   source_id : 2 stream_name MOJD2
29.89 (32.10)   source_id : 1 stream_name MOJD1
29.49 (32.39)   source_id : 0 stream_name MOJD4

It specifies the number of sources for each pipeline, the current date/time, fps values, source_id’s, and stream names. The first number per line is the current fps value while the number in parenthesis is the average for that stream. source_id is an internal id DeepStream uses to keep track of each stream. This also corresponds to which tile the stream is displayed on in the DeepStream output overlay. stream_name is what is set via VST when the stream is initially added.

SDR logs are saved in the /data/logging-volume/sdr-deepstream.log file by default. Live logs can be viewed using the command tail -f /data/logging-volume/sdr-deepstream.log. The current streams added to DeepStream via SDR can be viewed in this file as well as the latest relevant messages SDR picks up from VST over Redis.