Video Storage Toolkit#

Introduction#

Video Storage Toolkit (VST) also referred as VMS (Video Management System) manages audio and video streams and provides on demand access to offline streams from storage. It accepts WebRTC stream from front end UI application and outputs RTSP streams for further processing. It is also responsible for streaming out Avatar animation stream to front end UI application. VST effectively manages stream quality which is suitable for the current available bandwidth. It provides low bandwidth events to front end UI using WebRTC data channel. VST also provides API to download video clips with audio of user session for offline analysis.

Tokkio vst overview Diagram

  • Input:

    1. Webcam stream audio and/or video from browser client;

    2. Avatar udp stream provided by OV-renderers

  • Output:

    1. Avatar Webrtc output stream rendered on browser client;

    2. RTSP stream to botmaker (audio) & DS (video)

    3. Redis notifications to downstream services notify about new webcam-stream/session

Features#

  1. Media streaming using hardware-accelerated WebRTC protocol for live and recorded videos.

  2. RTSP streaming in pass-through and multi-cast mode.

  3. RTP-UDP streaming.

  4. Video storage and recording with aging policy.

  5. Supports both H.264 and H.265 video formats.

  6. Manage devices manually by IP address and/or RTSP URLs.

  7. Provide REST APIs to write client application to control and configure VST.

  8. Support Redis message bus to publish device add/remove events.

  9. Prometheus/Grafana integration for publishing VST statistics.

  10. Cloud native. Deploy as a container.

VST Architecture#

VST architecture

UI interaction#

Webcam stream client interaction

VST webcam streaming sequence

Avatar stream client interaction

VST avatar streaming sequence

API Guide#

POST: api/v1/streambridge/iceCandidate

Description: Add new remote candidate to the RTCPeerConnection's remote description
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
  "candidate": {
    "candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
    "sdpMLineIndex": 0,
    "sdpMid": "0"
  }
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

POST: api/v1/streambridge/stream/start

Description: Start webrtc streaming
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
  "isClient" : true, (optional)
  "isDataChannel" : true, (optional)
  "options": {
    "rtptransport": "udp",
    "timeout": 60,
  },
  "sessionDescription": {
    "type": "offer",
    "sdp": <sdp-offer>
  }
}
Response Schema:
200
{
  "sdp": <sdp-answer>,
  "type": "answer"
}
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

isClient should be set to true when VST is in receiver mode. If it is not set or set to false then VST will be in sender mode.

isDataChannel should be set to true when client want to establish data channel with VST. The network quality information is shared by VST on data channel.

POST: api/v1/streambridge/stream/stop

Description: Stop webrtc streaming
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b"
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

POST: api/v1/streambridge/setAnswer

Description: Set Webrtc re-negotiation answer
Query parameters:
peerId
Request Schema:
{
  "sessionDescription": {
    "type": "answer",
    "sdp": <sdp-answer>
  }
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/iceCandidate

Description: Add new remote candidate to the RTCPeerConnection's remote description
Query parameters:
peerId
Response Schema:
200
[
  {
    "candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
    "sdpMLineIndex": 0,
    "sdpMid": "0"
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/iceServers

Description: Get URL or URLs of the servers to be used for ICE negotiations. These are typically STUN and/or TURN servers
Query parameters:
peerId
Response Schema:
200
{
  "iceServers": [
    {
      "urls": [
        "stun:stun.l.google.com:19302"
      ]
    }
  ]
}
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/sensor/list

Description: Get list of devices with some details
Response Schema:
200
[
  {
    "firmwareVersion" : "",
    "hardware" : "",
    "hardwareId" : "",
    "isRemoteSensor" : false,
    "location" : "",
    "manufacturer" : "",
    "name" : "webcam_stream",
    "position" :
    {
      "coordinates" :
      {
        "x" : "",
        "y" : ""
      },
      "depth" : "",
      "direction" : "",
      "fieldOfView" : "",
      "geoLocation" :
      {
        "latitude" : "",
        "longitude" : ""
      },
      "origin" :
      {
        "latitude" : "",
        "longitude" : ""
      }
    },
    "remoteDeviceId" : "929696fc-4232-479a-a582-5898d501857e",
    "remoteDeviceLocation" : "Pune",
    "remoteDeviceName" : "VST",
    "sensorId" : "4d13e7d6-6b1b-44ab-b50c-a6b208e26941",
    "sensorIp" : "",
    "serialNumber" : "",
    "state" : "online",
    "tags" : ""
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/streams

Description: Get list of available streambridge streams
Response Schema:
200
[
  {
    "371af362-07ae-498f-bab9-84d851353e94": [
      {
        "isMain": true,
        "metadata": {
          "bitrate": "",
          "codec": "h264",
          "framerate": "30",
          "govlength": "",
          "resolution": "1920x1080"
        },
        "name": "webcam_stream",
        "streamId": "371af362-07ae-498f-bab9-84d851353e94",
        "url": "rtsp://10.41.18.147:8554/webrtc/371af362-07ae-498f-bab9-84d851353e94"
      },
      {
        "isMain" : false,
        "metadata" : {
          "bitrate" : "",
          "codec" : "h264",
          "framerate" : "30",
          "govlength" : "",
          "resolution" : "1920x1080"
        },
        "name" : "Tokkio_Avatar_371af362-07ae-498f-bab9-84d851353e94_1",
        "streamId" : "371af362-07ae-498f-bab9-84d851353e94_1",
        "url" : ""
      }
    ]
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/file/{sensor-id}

Description: Download video files
Query parameters:
startTime
endTime
fullLength
Response Schema:
200
HTTP OK
<media-file>
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

WebSocket: vms/ws

Description: Receive offer from VST when new track available, on websocket connection
Query parameters:
connectionId
Response Schema:
{
  "type": "offer",
  "sdp": <sdp-offer>
}

VST Configuration#

The following VST configuration options can be modified in tokkio-app-params.yaml file. At least one valid turnserver (coturn/twilio) or valid reverse-proxy server should be provided for webcam and avatar streaming to work over webRTC

VST Configuration#

stunurl_list

set list of stun URLs, It discovers their public IP and type of NAT

[“stun.l.google.com:19302”,”stun1.l.google.com:19302”]

static_turnurl_list

list of TURN servers with long term credential mechanism. TURN provides a relay mechanism for communication when direct peer-to-peer communication is not possible due to NAT traversal issues.

[“admin:admin@10.0.0.1:3478”, “admin:admin@10.0.0.1:3478”]

use_coturn_auth_secret

enable authentication secret mechanism for coturn

false

coturn_turnurl_list_with_secret

list of co-turn servers with short term credential mechanism. To use this config, use_coturn_auth_secret should be set to true

[“10.0.0.1:3478”:<secret_key>, “10.0.0.1:3478”:<secret_key>]

use_twilio_stun_turn

enable to use twilio stun and turn server

false

twilio_account_sid

twilio account username. The config option use_twilio_stun_turn should be set to true to enable use of twilio server

“”

twilio_auth_token

authentication token of twilio account

“”

use_reverse_proxy

use reverse proxy instead of turnserver (coturn/twilio). RP is public facing service that directly receives and handles client traffic, performing the appropriate routing so that the traffic arrives at tokkio cluster VPC.

false

reverse_proxy_server_address

if use_reverse_proxy is set to true, then set reverse_proxy server address & port

10.0.0.1:100

max_webrtc_out_connections

set maximum count of webrtc out connections i.e avatar stream

8

max_webrtc_in_connections

set maximum count of webrtc in connections i.e webcam stream

3

enable_grpc

enable GRPC server used to port negotiate with ov-renderer to receive avatar udp stream

true

grpc_server_port

set GRPC server port

50051

udp_latency_ms

set udp stream latency buffer size

200

udp_drop_on_latency

drop udp packets on exceeding latency buffer

false

webrtc_in_audio_sender_max_bitrate

set max bitrate in kbps to be used by web-UI for webcam stream

128000

webrtc_in_video_degradation_preference

set degradation preference to be used by webrtc sender for webcam stream. It controls the quality of media streams (either resolution or framerate) when network conditions degrade

“resolution”

webrtc_in_video_sender_max_framerate

set max framerate to be used to by webrtc sender for webcam stream

30

webrtc_in_video_bitrate_thresold_percentage

set threshold percentage for video bitrate at which VST will alert client about low video quality. Standard bitrate is H264 720p@30 at 3mbps

50

total_video_storage_size_MB

set max video record size

10000

always_recording

set always recording on or off

false

webrtc_in_fixed_resolution

set fixed resolution for webcam stream

1280x720

enable_notification

enable redis notification, it is used to notify downstream services when there is new session/webcam stream

true

use_message_broker

use message broker either redis or kafka

redis

message_broker_topic

set message broker topic name

vst_events

gpu_indices

set GPU indices to use particular gpu device

[]

webrtc_port_range

set webrtc min and max port range

min 30001, max 30030

use_software_path

enable or disable software path, if gpu is not available.

false

use_https

enable or disable HTTPS

false

use_http_digest_authentication

enable or disable digest authentication

false

use_multi_user

enable or disable VST multi user feature

false

multi_user_extra_options

add extra options in header

[“Secure”, “SameSite=none”]

enable_user_cleanup

enable or disable VST user cleanup on device deletion

true

session_max_age_sec

set VST session max age

2592000
On this page