Client App - Reference Cloud - Device Call flows

Introduction

The Metropolis Microservices on the Jetson platform comprise a suite of microservices, each offering APIs for configuration and interaction purposes. This section offers an overview of the call flows associated with key features utilized by the mobile application. It serves as a guide for users who wish to develop their custom clients. Although comprehensive API definitions are available in the respective module documentations, this section focuses on a concise summary of the major callflows associated with Metropolis Microservices.

video streaming from VST
tripwire and analytics gem creation
cloud based device API invocation

Note that device API invocation through direct access instead of the cloud would follow identical callflows, with the device ingress endpoint being targeted at <device ip>:30080 instead of the the cloud based endpoints.

This section illustrates the call flows related to the use of VST and eMDAT APIs within the overall system. For detailed information on the specific API signatures, please refer to the respective sections dedicated to the VST and eMDAT APIs.

client - AWS Cognito (IDP) Auth call flow

This section captures the Authentication call flow between a client and AWS Cognito. Call flow will be similar in case if any other OAuth 2.0 standard IdP provider is used for Authentication.

Mobile client sends an authorization request to AWS Cognito by logging in from the IdP hosted webpage.
AWS Cognito sends back authorization grant in the form of an authorization code embedded in the url while redirecting the client back to a predefined URL.
Mobile client requests the user profile at AWS Cognito with the received authorization grant.
AWS Cognito returns a ID token back that captures the user profile and IdP information, signed by IdP private key.
Mobile client uses the id token to invoke API calls at the cloud proxy.
Cloud proxy validates the authenticity and permission of the id token, returns the protected resource back to the mobile client.

Stream enumeration from VST and addition to DeepStream

We start with the callflow most fundamental to MoJ - stream discovery and inference.

This callflow is fundamental to the SDR microservice operation, where streams from the VST microservice are added to DeepStream using its stream addition API

On startup, SDR fetches current authenticated streams from the VST v1/live/streams API endpoint. This returns a list of streams in the following format:

[
    {
        "44baa78d-76be-4ee9-aed5-da16c24edfec" :
        [
            {
                "isMain" : true,
                "metadata" :
                {
                    "bitrate" : "8000",
                    "codec" : "H264",
                    "framerate" : "30",
                    "govlength" : "60",
                    "resolution" : "1920x1080"
                },
                "name" : "Amcrest",
                "streamId" : "44baa78d-76be-4ee9-aed5-da16c24edfec",
                "url" : "rtsp://172.17.169.78/live/44baa78d-76be-4ee9-aed5-da16c24edfec"
            },
            {
                "isMain" : false,
                "metadata" :
                {
                    "bitrate" : "512",
                    "codec" : "H264",
                    "framerate" : "15",
                    "govlength" : "30",
                    "resolution" : "704x480"
                },
                "name" : "Amcrest-MediaProfile_Channel1_SubStream1",
                "streamId" : "44baa78d-76be-4ee9-aed5-da16c24edfec-MediaProfile00001",
                "url" : "rtsp://172.17.169.78/live/44baa78d-76be-4ee9-aed5-da16c24edfec-MediaProfile00001"
            }
        ]
    },
    {
        "a2c44a42-5a7f-4c7c-bed1-b2c6841d9350" :
        [
            {
                "isMain" : true,
                "metadata" :
                {
                    "bitrate" : "",
                    "codec" : "H264",
                    "framerate" : "",
                    "govlength" : "",
                    "resolution" : ""
                },
                "name" : "MOJD1",
                "streamId" : "a2c44a42-5a7f-4c7c-bed1-b2c6841d9350",
                "url" : "rtsp://172.17.169.78/live/a2c44a42-5a7f-4c7c-bed1-b2c6841d9350"
            }
        ]
    }
]

This JSON is parsed and for each value in the array, only the ones marked as “isMain”: true and with non-empty url values are then added to DeepStream.

After this initial check, SDR listens to Redis events to see if any new streams are added in VST. Each time a stream is added and becomes ready for streaming in VST, VST sends an event to redis with the following format:

{
    "alert_type": "camera_status_change",
    "created_at": "2023-11-20T23:51:04Z",
    "event": {
        "camera_id": "MOJD7",
        "camera_name": "36398191-d48a-4260-b574-239fd59f156f",
        "camera_url": "rtsp://172.17.170.143/live/36398191-d48a-4260-b574-239fd59f156f",
        "change": "camera_streaming"
    },
    "source": "vst"
}

Similarly, remove events are also sent from VST via Redis:

{
    "alert_type": "camera_status_change",
    "created_at": "2023-11-20T23:51:04Z",
    "event": {
        "camera_id": "MOJD7",
        "camera_name": "36398191-d48a-4260-b574-239fd59f156f",
        "camera_url": "rtsp://172.17.170.143/live/36398191-d48a-4260-b574-239fd59f156f",
        "change": "camera_remove"
    },
    "source": "vst"
}

Each time a new stream event is sent on Redis or picked up from the VST v1/live/streams endpoint, SDR parses it to determine if it should be ignored, if a stream should be added, or if a stream should be removed. Events are ignored if for instance the change event is not one SDR is configured to listen to (by default camera_streaming and camera_remove), if some required value is missing (camera_id, camera_name, camera_url), or if the name matches a regex value (by default this checks to make sure the name doesn’t include the substring overlay (ignoring capitalization)).

In order to add and remove streams from DeepStream, SDR uses the add/remove API implemented in DeepStream. This API is enabled when using [source-list] and [source-attr-all] in the DeepStream config rather than single sources (ie. [source0], [source1], etc.,).

To add a stream to DeepStream, a POST request must be made to the http://localhost:9010/api/v1/stream/add endpoint. Note that localhost may need to be changed to the proper IP and the port from 9010 to something else - by default port 9010 is used for pipeline 1 and port 9011 for pipeline 2, this is specified in the DeepStream config files. The following JSON must also be included in the request:

{
    "key": "sensor",
    "value": {
        "camera_id": "Amcrest_1",
        "camera_name": "Amcrest_1",
        "camera_url": "rtsp://10.136.27.7/live/7995217a-3f05-47a5-885d-50c937bf2ed5",
        "change": "camera_add",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}

To remove a stream from DeepStream, a POST request must be made to the http://localhost:9010/api/v1/stream/remove endpoint. Again, localhost may need to be changed to the proper IP and specify the correct port. The following JSON must also be included in the request:

{
    "key": "sensor",
    "value": {
        "camera_id": "Amcrest_1",
        "camera_name": "Amcrest_1",
        "camera_url": "rtsp://10.136.27.7/live/7995217a-3f05-47a5-885d-50c937bf2ed5",
        "change": "camera_remove",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}

Note that only the camera_id, camera_name, camera_url, and change values are necessary, everything else is optional.

Video Streaming using WebRTC from VST

WebRTC (Web Real-Time Communication) is a technology enabling real-time communication such as audio, video, and data transfer directly in web browsers and mobile applications. It facilitates peer-to-peer connections, bypassing the need for additional plugins. Key components in WebRTC include MediaStream for accessing camera and microphone, RTCPeerConnection for maintaining direct connections, and RTCDataChannel for data transfer. The protocol suite ensures efficient, secure, and versatile communication in real-time.

Refer to - https://webrtc.org/getting-started/overview to get started with WebRTC.

User can follow through the WebRTC samples documented here to understand the steps involved in building a typical WebRTC peer in a video streaming. WebRTC samples - https://github.com/webrtc/samples . Apart from WebRTC signaling APIs shown below, building a WebRTC peer involves calling standard WebRTC APIs as captured in the WebRTC samples.

VST running on the device uses WebRTC as a default protocol for video streaming. Below sequence diagram captures details on the actual VST APIs a client app can use while implementing a WebRTC client to stream video from VST. APIs shown in the sequence are custom WebRTC signaling APIs used by VST to share STUN/TURN server details for NATing and exchange ICE candidate details between the client application and VST.

Tripwire and ROI Gem creation and usage

Creating and use tripwires and ROI is another core functionality used by client applications.

These APIs use the sensor names retrieved using VST APIs to create tripwire/ROI gems using HTTP POST requests.

Coordinates for installing these are specified based pixel positions in the video field of view based on camera resolution.

Creation
Below call flow captures the APIs used by a client app to create a new Tripwire.

Query
Below call flow captures the APIs used by a client app to query a Tripwire metrics

ROI Gem

Creation

Below call flow captures the APIs used by a client to create a new ROI.

Query

Below call flow captures the APIs used by a client to query a ROI metrics