Video Storage Toolkit#

Introduction#

Video Storage Toolkit (VST) also referred as VMS (Video Management System) manages audio and video streams and provides on demand access to offline streams from storage. It accepts WebRTC stream from front end UI application and outputs RTSP streams for further processing. It is also responsible for streaming out Avatar animation stream to front end UI application. VST effectively manages stream quality which is suitable for the current available bandwidth. It provides low bandwidth events to front end UI using WebRTC data channel. VST also provides API to download video clips with audio of user session for offline analysis.

Tokkio vst overview Diagram

  • Input:

    1. Webcam stream audio and/or video from browser client;

    2. Avatar udp stream provided by OV-renderers

  • Output:

    1. Avatar Webrtc output stream rendered on browser client;

    2. RTSP stream to Chat Controller (audio) & DS (video)

    3. Redis notifications to downstream services notify about new webcam-stream/session

Features#

  1. Media streaming using hardware-accelerated WebRTC protocol for live and recorded videos.

  2. RTSP streaming in pass-through and multi-cast mode.

  3. RTP-UDP streaming.

  4. Video storage and recording with aging policy.

  5. Supports both H.264 and H.265 video formats.

  6. Manage devices manually by IP address and/or RTSP URLs.

  7. Provide REST APIs to write client application to control and configure VST.

  8. Support Redis message bus to publish device add/remove events.

  9. Prometheus/Grafana integration for publishing VST statistics.

  10. Cloud native. Deploy as a container.

VST Architecture#

VST architecture

UI interaction#

Webcam stream client interaction

VST webcam streaming sequence

Avatar stream client interaction

VST avatar streaming sequence

API Guide#

POST: api/v1/streambridge/iceCandidate

Description: Add new remote candidate to the RTCPeerConnection's remote description
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
  "candidate": {
    "candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
    "sdpMLineIndex": 0,
    "sdpMid": "0"
  }
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

POST: api/v1/streambridge/stream/start

Description: Start webrtc streaming
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
  "isClient" : true, (optional)
  "isDataChannel" : true, (optional)
  "options": {
    "rtptransport": "udp",
    "timeout": 60,
  },
  "sessionDescription": {
    "type": "offer",
    "sdp": <sdp-offer>
  }
}
Response Schema:
200
{
  "sdp": <sdp-answer>,
  "type": "answer"
}
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

isClient should be set to true when VST is in receiver mode. If it is not set or set to false then VST will be in sender mode.

isDataChannel should be set to true when client want to establish data channel with VST. The network quality information is shared by VST on data channel.

POST: api/v1/streambridge/stream/stop

Description: Stop webrtc streaming
Request Schema:
{
  "peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b"
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

POST: api/v1/streambridge/setAnswer

Description: Set Webrtc re-negotiation answer
Query parameters:
peerId
Request Schema:
{
  "sessionDescription": {
    "type": "answer",
    "sdp": <sdp-answer>
  }
}
Response Schema:
200
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/iceCandidate

Description: Add new remote candidate to the RTCPeerConnection's remote description
Query parameters:
peerId
Response Schema:
200
[
  {
    "candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
    "sdpMLineIndex": 0,
    "sdpMid": "0"
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/iceServers

Description: Get URL or URLs of the servers to be used for ICE negotiations. These are typically STUN and/or TURN servers
Query parameters:
peerId
Response Schema:
200
{
  "iceServers": [
    {
      "urls": [
        "stun:stun.l.google.com:19302"
      ]
    }
  ]
}
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/sensor/list

Description: Get list of devices with some details
Response Schema:
200
[
  {
    "firmwareVersion" : "",
    "hardware" : "",
    "hardwareId" : "",
    "isRemoteSensor" : false,
    "location" : "",
    "manufacturer" : "",
    "name" : "webcam_stream",
    "position" :
    {
      "coordinates" :
      {
        "x" : "",
        "y" : ""
      },
      "depth" : "",
      "direction" : "",
      "fieldOfView" : "",
      "geoLocation" :
      {
        "latitude" : "",
        "longitude" : ""
      },
      "origin" :
      {
        "latitude" : "",
        "longitude" : ""
      }
    },
    "remoteDeviceId" : "929696fc-4232-479a-a582-5898d501857e",
    "remoteDeviceLocation" : "Pune",
    "remoteDeviceName" : "VST",
    "sensorId" : "4d13e7d6-6b1b-44ab-b50c-a6b208e26941",
    "sensorIp" : "",
    "serialNumber" : "",
    "state" : "online",
    "tags" : ""
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/streambridge/streams

Description: Get list of available streambridge streams
Response Schema:
200
[
  {
    "371af362-07ae-498f-bab9-84d851353e94": [
      {
        "isMain": true,
        "metadata": {
          "bitrate": "",
          "codec": "h264",
          "framerate": "30",
          "govlength": "",
          "resolution": "1920x1080"
        },
        "name": "webcam_stream",
        "streamId": "371af362-07ae-498f-bab9-84d851353e94",
        "url": "rtsp://10.41.18.147:8554/webrtc/371af362-07ae-498f-bab9-84d851353e94"
      },
      {
        "isMain" : false,
        "metadata" : {
          "bitrate" : "",
          "codec" : "h264",
          "framerate" : "30",
          "govlength" : "",
          "resolution" : "1920x1080"
        },
        "name" : "Tokkio_Avatar_371af362-07ae-498f-bab9-84d851353e94_1",
        "streamId" : "371af362-07ae-498f-bab9-84d851353e94_1",
        "url" : ""
      }
    ]
  }
]
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

GET: api/v1/file/{sensor-id}

Description: Download video files
Query parameters:
startTime
endTime
fullLength
Response Schema:
200
HTTP OK
<media-file>
Error Schema:
{
  "error_code": "VMSInternalError",
  "error_message": "VMS internal processing error"
}

WebSocket: vms/ws

Description: Receive offer from VST when new track available, on websocket connection
Query parameters:
connectionId
Response Schema:
{
  "type": "offer",
  "sdp": <sdp-offer>
}

VST Configuration#

The following VST configuration options can be modified in tokkio-app-params.yaml file. At least one valid turnserver (coturn/twilio) or valid reverse-proxy server should be provided for webcam and avatar streaming to work over webRTC

VST Configuration#

stunurl_list

set list of stun URLs, It discovers their public IP and type of NAT

[“stun.l.google.com:19302”,”stun1.l.google.com:19302”]

static_turnurl_list

list of TURN servers with long term credential mechanism. TURN provides a relay mechanism for communication when direct peer-to-peer communication is not possible due to NAT traversal issues.

[“admin:admin@10.0.0.1:3478”, “admin:admin@10.0.0.1:3478”]

use_coturn_auth_secret

enable authentication secret mechanism for coturn

false

coturn_turnurl_list_with_secret

list of co-turn servers with short term credential mechanism. To use this config, use_coturn_auth_secret should be set to true

[“10.0.0.1:3478”:<secret_key>, “10.0.0.1:3478”:<secret_key>]

use_twilio_stun_turn

enable to use twilio stun and turn server

false

twilio_account_sid

twilio account username. The config option use_twilio_stun_turn should be set to true to enable use of twilio server

“”

twilio_auth_token

authentication token of twilio account

“”

use_reverse_proxy

use reverse proxy instead of turnserver (coturn/twilio). RP is public facing service that directly receives and handles client traffic, performing the appropriate routing so that the traffic arrives at tokkio cluster VPC.

false

reverse_proxy_server_address

if use_reverse_proxy is set to true, then set reverse_proxy server address & port. Also env variable REVERSE_PROXY_SERVER_ADDRESS can be used to set the ip_address.

10.0.0.1:100

max_webrtc_out_connections

set maximum count of webrtc out connections i.e avatar stream

8

max_webrtc_in_connections

set maximum count of webrtc in connections i.e webcam stream

3

grpc_server_port

set GRPC server port. GRPC server used to port negotiate with ov-renderer to receive avatar udp stream

50051

webrtc_in_audio_sender_max_bitrate

set max bitrate in kbps to be used by web-UI for webcam stream

128000

webrtc_in_video_degradation_preference

set degradation preference to be used by webrtc sender for webcam stream. It controls the quality of media streams (either resolution or framerate) when network conditions degrade

“framerate”

total_video_storage_size_MB

set max video record size used to record webcam stream

10000

always_recording

set always recording on or off

false

gpu_indices

set GPU indices to use particular gpu device inside vst container, If there are multiple gpu devices visible inside container

[]

webrtc_port_range

set webrtc min and max port range. This should be in sync with nodePort range in the VST microservice helm-chart. VST uses nodeports for webrtc media traffic due to separate RP instance

min 30001, max 30030

use_software_path

enable or disable software path, if gpu is not available.

false

enable_websocket_pingpong

enable websocket periodic ping pong.This is to avoid websocket connection break due to proxy/LB or any firewall policies

false

websocket_keep_alive_ms

websocket periodic ping pong time in milliseconds

5000
On this page