Face Detection and Tracking#

Overview#

The main process is a Video Inference Service consists of face detection, object tracker, message producer, etc. It uses HTTP API for adding or removing input video streams dynamically.

The secondary process is a Controller Sidecar that drives the HTTP API call to Video Inference Service. It keeps querying current live video stream URLs from Audio-Video Stream Provider, then compare with current processing video stream URL in Video Inference Service. Finally query Video Inference Service to update input video stream, eg removing old one and add new one.

Architecture_Diagram

  • Input:

    1. Audio-Video Stream Provider endpoint;

    2. Stable video stream in RTSP provided by Audio-Video Stream Provider

  • Output:

    1. Message stream of per frame face bounding box;

  • Supported Platforms: x86 dGPU

  • Deployment requirements: Make sure k8s foundational services are running

  • Features:

    1. Face Detect

    2. Dynamic input video stream update

    3. Automatic input video stream reconnect upon lossing data for certain time

  • Algorithms:

    Video Inference Service

    1. Implements HTTP service to add and remove input video streams dynamically

    2. Face Detection produces face label and bounding box coordinates

    3. Tracker produces tracking ID to indicate the same face object

    4. Each video stream has its unique ID from Audio-Video Stream Provider, the same ID will be associated with metadata in output message stream

    Controller Sidecar process

    1. Uses Audio-Video Stream Provider’s client and Video Inference Service client to query running streams periodically, eg default interval is 15 seconds

    2. Compare running video streams list from Video Inference Service and Audio-Video Stream Provider, and find out the delta

    3. For each interval, apply the delta to Video Inference Service by removing old streams no longer in Audio-Video Stream Provider, and new streams just appear in Audio-Video Stream Provider

Usage#

Sample Params:

ds-visionai:
  rtspReconnectInterval: 10
  redisCfg:
    payloadkey: metadata
    topic: "visionai"
  controllerSidecar:
    recessPeriod: "15s"

Sample Connections:

connections:
  ds-visionai/vms: vms/vms
  ds-visionai/redis: redis/redis
  <DOWNSTREAM_BBOX_ANALYTICS_MS>/redis: redis/redis

Note

In this example, vms is the name of microservice Audio-Video Stream Provider; ds-visionai is the name of microservice Face Detection and Tracking.

vms –> http(for input stream URL) –> ds-visionai(Controller Sidecar)

vms –> RTSP(for input stream) –> ds-visionai(Video Inference Service)

ds-visionai –> redis –> <DOWNSTREAM_BBOX_ANALYTICS_MS>

All Parameters:

Face Detection and Tracking Params#

Param

Description

dsServicePort: (integer )

DS service port. Default 8084

remoteAccess: (string )

Enable remote access or localhost only. Default ‘True’

addStreamApiPath: (string )

API path for service to add a stream. Default AddStream

removeStreamApiPath: (string )

API path for service to remove a stream. Default RemoveStream

getStatusApiPath: (string )

API path for service to get status. Default Status

apiResourceName: (string )

REST API resource name for stream. Default stream

recycleSourceId: (string )

Recycle and reuse DS internal source-id upon removing a stream. Default ‘False’ [Mandatory:False, Allowed Values:{ True, False }]

maxNumSources: (integer )

max number of sources allowed to add. Default 6

batchSize: (integer )

batch size for streammux and inference. Default 8

rtspReconnectInterval: (integer )

Timeout in seconds to wait since last data was received from an RTSP source before forcing a reconnection. 0=disable timeout. Default 10

rtpProtocol: (integer )

RTP protocol to use. 0 for TCP/UDP; 4 for TCP Only. Default 0 [Mandatory:False, Allowed Values:{ 0, 4 }]

peerPadIdSameAsSourceId: (string )

Use id comes from source stream in streammux stream index. Default ‘True’ [Mandatory:False, Allowed Values:{ true, false }]

jitterbufferLatency: (integer )

Pipeline source component Jitterbuffer size in milliseconds. Default 100 [Mandatory:False]

fileLoop: (string )

loop a file input. Default ‘True’ [Mandatory:False, Allowed Values:{ true, false }]

msgConvPayloadType: (integer )

DS msgbroker payload schema. 0 DS; 1 DS minimal; 256 Reserved; 257 Custom. Default 1 [Mandatory:False, Allowed Values:{ 0, 1, 256, 257 }]

redisCfg: (object )

Redis broker config

payloadkey: (string )

Payload key for messages. Default metadata

topic: (string )

topic or stream name for redis messages. Default ‘test’

enableLatency: (string )

Whether to enable per frame latency measure. Default ‘false’ [Mandatory:False, Allowed Values:{ true, false }]

enableCompLatency: (string )

Whether to enable per component latency; only used when enableLatency is ‘True’. Default ‘False’ [Mandatory:False, Allowed Values:{ true, false }]

videoSink: (string )

type of video sink. Default ‘none’ [Mandatory:False, Allowed Values:{ rtsp, fake, file, none }]

sinkSync: (string )

pipeline sink component sync on the clock. Default ‘false’ [Mandatory:False, Allowed Values:{ true, false }]

controllerSidecar: (object )

service controller sidecar config

recessPeriod: (string )

service controller operate interval in seconds. Default value is 15s

vmsEndpointHttps: (string )

Use http or https to setup vms http client. Default ‘False’ [Mandatory:False, Allowed Values:{ True, False }]

vmsGetStreamsApi: (string )

vms get streams API. Default ‘api/device/streams’

queryStreamFromVms: (string )

Whether to query VMS for an RTSP stream. Default ‘True’ [Mandatory:False, Allowed Values:{ true, false }]

streammuxResolution: (object )

DS input video resolution config

height: (integer )

expected video frame height. Default 1080

width: (integer )

expected video frame width. Default 1920

checkInterval: (string )

init container check interval in seconds. Default ‘1’

On this page