Video Storage Toolkit
Introduction
Video Storage Toolkit (VST) also referred as VMS (Video Management System) manages audio and video streams and provides on demand access to offline streams from storage. It accepts WebRTC stream from front end UI application and outputs RTSP streams for further processing. It is also responsible for streaming out Avatar animation stream to front end UI application. VST effectively manages stream quality which is suitable for the current available bandwidth. It provides low bandwidth events to front end UI using WebRTC data channel. VST also provides API to download video clips with audio of user session for offline analysis.
Input:
Webcam stream audio and/or video from browser client;
Avatar udp stream provided by OV-renderers
Output:
Avatar Webrtc output stream rendered on browser client;
RTSP stream to botmaker (audio) & DS (video)
Redis notifications to downstream services notify about new webcam-stream/session
Features
Media streaming using hardware-accelerated WebRTC protocol for live and recorded videos.
RTSP streaming in pass-through and multi-cast mode.
RTP-UDP streaming.
Video storage and recording with aging policy.
Supports both H.264 and H.265 video formats.
Manage devices manually by IP address and/or RTSP URLs.
Provide REST APIs to write client application to control and configure VST.
Support Redis message bus to publish device add/remove events.
Prometheus/Grafana integration for publishing VST statistics.
Cloud native. Deploy as a container.
VST Architecture
UI interaction
Webcam stream client interaction
Avatar stream client interaction
API Guide
POST: api/v1/streambridge/iceCandidate
Description: Add new remote candidate to the RTCPeerConnection's remote description
Request Schema:
{
"peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
"candidate": {
"candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
"sdpMLineIndex": 0,
"sdpMid": "0"
}
}
Response Schema:
200
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
POST: api/v1/streambridge/stream/start
Description: Start webrtc streaming
Request Schema:
{
"peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b",
"isClient" : true, (optional)
"isDataChannel" : true, (optional)
"options": {
"rtptransport": "udp",
"timeout": 60,
},
"sessionDescription": {
"type": "offer",
"sdp": <sdp-offer>
}
}
Response Schema:
200
{
"sdp": <sdp-answer>,
"type": "answer"
}
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
isClient should be set to true when VST is in receiver mode. If it is not set or set to false then VST will be in sender mode.
isDataChannel should be set to true when client want to establish data channel with VST. The network quality information is shared by VST on data channel.
POST: api/v1/streambridge/stream/stop
Description: Stop webrtc streaming
Request Schema:
{
"peerId": "346c5ba8-eb8e-4239-9134-9ddf2f4adb5b"
}
Response Schema:
200
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
POST: api/v1/streambridge/setAnswer
Description: Set Webrtc re-negotiation answer
Query parameters:
peerId
Request Schema:
{
"sessionDescription": {
"type": "answer",
"sdp": <sdp-answer>
}
}
Response Schema:
200
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
GET: api/v1/streambridge/iceCandidate
Description: Add new remote candidate to the RTCPeerConnection's remote description
Query parameters:
peerId
Response Schema:
200
[
{
"candidate": "candidate:878186389 1 udp 2122260223 192.168.55.1 43519 typ host generation 0 ufrag IUnY network-id 2 network-cost 50",
"sdpMLineIndex": 0,
"sdpMid": "0"
}
]
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
GET: api/v1/streambridge/iceServers
Description: Get URL or URLs of the servers to be used for ICE negotiations. These are typically STUN and/or TURN servers
Query parameters:
peerId
Response Schema:
200
{
"iceServers": [
{
"urls": [
"stun:stun.l.google.com:19302"
]
}
]
}
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
GET: api/v1/sensor/list
Description: Get list of devices with some details
Response Schema:
200
[
{
"firmwareVersion" : "",
"hardware" : "",
"hardwareId" : "",
"isRemoteSensor" : false,
"location" : "",
"manufacturer" : "",
"name" : "webcam_stream",
"position" :
{
"coordinates" :
{
"x" : "",
"y" : ""
},
"depth" : "",
"direction" : "",
"fieldOfView" : "",
"geoLocation" :
{
"latitude" : "",
"longitude" : ""
},
"origin" :
{
"latitude" : "",
"longitude" : ""
}
},
"remoteDeviceId" : "929696fc-4232-479a-a582-5898d501857e",
"remoteDeviceLocation" : "Pune",
"remoteDeviceName" : "VST",
"sensorId" : "4d13e7d6-6b1b-44ab-b50c-a6b208e26941",
"sensorIp" : "",
"serialNumber" : "",
"state" : "online",
"tags" : ""
}
]
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
GET: api/v1/streambridge/streams
Description: Get list of available streambridge streams
Response Schema:
200
[
{
"371af362-07ae-498f-bab9-84d851353e94": [
{
"isMain": true,
"metadata": {
"bitrate": "",
"codec": "h264",
"framerate": "30",
"govlength": "",
"resolution": "1920x1080"
},
"name": "webcam_stream",
"streamId": "371af362-07ae-498f-bab9-84d851353e94",
"url": "rtsp://10.41.18.147:8554/webrtc/371af362-07ae-498f-bab9-84d851353e94"
},
{
"isMain" : false,
"metadata" : {
"bitrate" : "",
"codec" : "h264",
"framerate" : "30",
"govlength" : "",
"resolution" : "1920x1080"
},
"name" : "Tokkio_Avatar_371af362-07ae-498f-bab9-84d851353e94_1",
"streamId" : "371af362-07ae-498f-bab9-84d851353e94_1",
"url" : ""
}
]
}
]
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
GET: api/v1/file/{sensor-id}
Description: Download video files
Query parameters:
startTime
endTime
fullLength
Response Schema:
200
HTTP OK
<media-file>
Error Schema:
{
"error_code": "VMSInternalError",
"error_message": "VMS internal processing error"
}
WebSocket: vms/ws
Description: Receive offer from VST when new track available, on websocket connection
Query parameters:
connectionId
Response Schema:
{
"type": "offer",
"sdp": <sdp-offer>
}
VST Configuration
The following VST configuration options can be modified in tokkio-app-params.yaml file. At least one valid turnserver (coturn/twilio) or valid reverse-proxy server should be provided for webcam and avatar streaming to work over webRTC
VST Configuration stunurl_list
set list of stun URLs, It discovers their public IP and type of NAT
[“stun.l.google.com:19302”,”stun1.l.google.com:19302”]
static_turnurl_list
list of TURN servers with long term credential mechanism. TURN provides a relay mechanism for communication when direct peer-to-peer communication is not possible due to NAT traversal issues.
[“admin:admin@10.0.0.1:3478”, “admin:admin@10.0.0.1:3478”]
use_coturn_auth_secret
enable authentication secret mechanism for coturn
false
coturn_turnurl_list_with_secret
list of co-turn servers with short term credential mechanism. To use this config, use_coturn_auth_secret should be set to true
[“10.0.0.1:3478”:<secret_key>, “10.0.0.1:3478”:<secret_key>]
use_twilio_stun_turn
enable to use twilio stun and turn server
false
twilio_account_sid
twilio account username. The config option use_twilio_stun_turn should be set to true to enable use of twilio server
“”
twilio_auth_token
authentication token of twilio account
“”
use_reverse_proxy
use reverse proxy instead of turnserver (coturn/twilio). RP is public facing service that directly receives and handles client traffic, performing the appropriate routing so that the traffic arrives at tokkio cluster VPC.
false
reverse_proxy_server_address
if use_reverse_proxy is set to true, then set reverse_proxy server address & port
10.0.0.1:100
max_webrtc_out_connections
set maximum count of webrtc out connections i.e avatar stream
8
max_webrtc_in_connections
set maximum count of webrtc in connections i.e webcam stream
3
enable_grpc
enable GRPC server used to port negotiate with ov-renderer to receive avatar udp stream
true
grpc_server_port
set GRPC server port
50051
udp_latency_ms
set udp stream latency buffer size
200
udp_drop_on_latency
drop udp packets on exceeding latency buffer
false
webrtc_in_audio_sender_max_bitrate
set max bitrate in kbps to be used by web-UI for webcam stream
128000
webrtc_in_video_degradation_preference
set degradation preference to be used by webrtc sender for webcam stream. It controls the quality of media streams (either resolution or framerate) when network conditions degrade
“resolution”
webrtc_in_video_sender_max_framerate
set max framerate to be used to by webrtc sender for webcam stream
30
webrtc_in_video_bitrate_thresold_percentage
set threshold percentage for video bitrate at which VST will alert client about low video quality. Standard bitrate is H264 720p@30 at 3mbps
50
total_video_storage_size_MB
set max video record size
10000
always_recording
set always recording on or off
false
webrtc_in_fixed_resolution
set fixed resolution for webcam stream
1280x720
enable_notification
enable redis notification, it is used to notify downstream services when there is new session/webcam stream
true
use_message_broker
use message broker either redis or kafka
redis
message_broker_topic
set message broker topic name
vst_events
gpu_indices
set GPU indices to use particular gpu device
[]
webrtc_port_range
set webrtc min and max port range
min 30001, max 30030
use_software_path
enable or disable software path, if gpu is not available.
false
use_https
enable or disable HTTPS
false
use_http_digest_authentication
enable or disable digest authentication
false
use_multi_user
enable or disable VST multi user feature
false
multi_user_extra_options
add extra options in header
[“Secure”, “SameSite=none”]
enable_user_cleanup
enable or disable VST user cleanup on device deletion
true
session_max_age_sec
set VST session max age
2592000