VST Troubleshooting#

Debugging using VST logs#

  • VST config: VST config is printed at the beginning of the VST logs. It can be used to check the current config with which VST is running.

Host HTTP port: 30000 Recorded Video Root: /home/vst/vst_release/vst_video stunurl list: stun.l.google.com:19302, stun1.l.google.com:19302 static_turnurl_list: use_coturn_auth_secret: 0 use_twilio_stun_turn: 0 coturn_turnurl_list_with_secret: twilio_account_sid: twilio_auth_token: use reverse proxy RP: 0

...

  • Redis server connection: This log can be used to check if VST is able to connect to the Redis server.

[2024-12-20T9:49:40.728Z:137990895721664:redis_publisher.cpp:176: redis_init] Redis server address:port= 127.0.0.1;6379 [2024-12-20T9:49:40.728Z:137990895721664:redis_publisher.cpp:187: redis_init] Redis connect success.

  • GRPC server is listening: This log can be used to check if the VST GRPC server is enabled and listening.

[2024-12-20T9:49:40.853Z:137988786882112:nvgrpc.cpp:931: RunServer]     gRPC Server listening on 0.0.0.0:50051

  • GPU detected: This log can be used to check if VST detected the GPU

[2024-12-20T9:49:40.557Z:137990895721664:utils.cpp:2227: detectGPU]     ############## GPU ID DETECTED = 0 ##############

[2024-12-20T9:49:40.557Z:137990895721664:utils.cpp:2228: detectGPU]     ############## GPU Device = /dev/nvidia0 ##############

  • Checking WebSocket Connection: This log can be used to check if the WebSocket connection was successful

[2024-12-20T9:50:42.164Z:137990862919232:Websocket.cpp:51: addConnection]       Saving connection ID: 1380cd96-aa97-41b4-8f4f-f64f00fc6568

[2024-12-20T9:50:42.164Z:137990862919232:Websocket.cpp:64: addConnection]       Memory allocated to websocket connection ID: 1380cd96-aa97-41b4-8f4f-f64f00fc6568

[2024-12-20T9:50:42.164Z:137990862919232:Websocket.cpp:66: addConnection]       Total websocket connections: 1

  • Checking WebCam/Microphone Peer Connections: To check if the Peer Connection of the WebCam or Microphone stream was successful, look for the logs below in VST logs.

[2024-12-20T9:50:45.223Z:137987255961152:WebrtcCallbacks.cpp:509: OnConnectionChange]    Peer connection state:Connected peerid:1380cd96-aa97-41b4-8f4f-f64f00fc6568

  • Checking if the Avatar Peer Connection between Tokkio UI and the UE service was successful; look for the logs below in the VST logs.

[2025-4-11T5:34:7.614Z:128275542464064:nvgrpc.cpp:542: remotePeerCandidate]    [2025-4-11T5:34:7.615Z:128275542464064:websocket_apis.cpp:224: operator()]      UE peerconnectionstatus : {
       "apiKey" : "/event/streambridge/ext/peerconnection/status",
       "data" :
       {
               "peerId" : "087f4fb6-ee5a-4272-8f66-40e3a6b9c9a1_1",
               "status" : "connected"
       },
       "peerId" : "087f4fb6-ee5a-4272-8f66-40e3a6b9c9a1_1"
}

  • Avatar & Microphone Redis events: This log indicates whether Redis events for the webcam stream were sent or not. There are two kinds of events: Camera add and Camera streaming. The logs below are relevant to check if Redis events are working or not.

[2024-12-20T9:50:43.485Z:137988304537152:PeerConnectionManager.cpp:2653: notify] { "alert_type" : "camera_status_change", "created_at" : "2024-12-20T09:50:43Z", "event" : { "camera_id" : "1380cd96-aa97-41b4-8f4f-f64f00fc6568", "camera_name" : "webcam_1380cd96-aa97-41b4-8f4f-f64f00fc6568", "camera_url" : "rtsp://10.41.25.10:30554/webrtc/1380cd96-aa97-41b4-8f4f-f64f00fc6568", "change" : "camera_add" }, "source" : "vst" }

[2024-12-20T9:50:43.485Z:137989615257152:redis_publisher.cpp:227: sendToRedis] Event sent to Redis Successfully

{ "alert_type" : "camera_status_change", "created_at" : "2024-12-20T09:50:45Z", "event" : { "camera_id" : "1380cd96-aa97-41b4-8f4f-f64f00fc6568", C1380cd96-aa97-41b4-8f4f-f64f00fc6568", "camera_url" : "rtsp://10.41.25.10:30554/webrtc/1380cd96-aa97-41b4-8f4f-f64f00fc6568", "change" : "camera_streaming" }, "source" : "vst" }

[2024-12-20T9:50:45.946Z:137989615257152:redis_publisher.cpp:227: sendToRedis] Event sent to Redis Successfully

  • FPS of avatar & microphone stream: The log below indicates the FPS of the microphone/webcam stream. Note that the FPS for the webcam will be displayed only when video is enabled. If only the microphone stream is enabled, then the FPS will not be printed. Check the sample logs below for more context. To get the FPS of the avatar stream, check the WebRTC stats in the web browser by going to the URL chrome://webrtc-internals/.

[2024-12-20T12:33:50.7490Z:137325361235520:fps_display.cpp:80: displayFPS] Unique ID = 7c4e2774-e3c2-415c-8c8b-ab3d420b92c9_1:7c4e2774-e3c2-415c-8c8b-ab3d420b92c9_in FPS = { 29.988873, 29.988873, 29.994843, 30.249584, 29.916251, 29.994792, 30.061507, 29.988936, 29.994792, 29.988936, 29.994843, 30.153816 }

WebRTC connection failed to connect.#

If the WebRTC port range is blocked in the deployed system, then both inbound and outbound WebRTC stream connections will fail. WebRTC requires a range of ports to be opened. Ensure that these ports are not blocked. The range is mentioned in the webrtc_port_range config of VST. Check if the firewall is not blocking the WebRTC ports. Flush the iptables to unblock the connection using the commands below.

sudo iptables -F
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

Outbound Stream (Microphone) is connected, but the inbound stream (Avatar) failed to connect.#

There can be many reasons why the inbound stream might fail to connect. The most common causes are related to the TURN server. Both peers should be able to generate relay ICE candidates on their side (not applicable in the case of RP or Reverse Proxy). If one of them is unable to generate relay candidates, then the inbound stream will fail to connect.

Resolution#

The TURN server needs to be properly configured and accessible. To check the functioning of the TURN server, we can use the methods below.

  1. Check VST & Tokkio UI logs to see if ICE Candidates of type relay are being generated. VST acts as a relay between Tokkio UI and the UE service. It forwards data from the UE service to Tokkio UI. So all WebRTC-related messages, like ICE Candidates, will be printed by VST as it receives them from the UE service.

For Tokkio UI logs put a “relay” string without quotes in the console logs filter. You should see a log similar to the below string indicating that a relay candidate was generated at the UI side.

onIceCandidate callback:  candidate:587694270 1 udp 1677729535 59.125.156.172 50276 typ relay raddr 10.60.186.82 rport 43969 generation 0 ufrag EQFc network-cost 50

For VST logs, search for string “relay” in the logs. You should see the log below indicating that the relay candidate was generated at UE side and forwarded to Tokkio UI by VST.

[2025-4-11T5:35:11.878Z:128270718731840:nvgrpc.cpp:715: operator()] candidate from grpc: {"sdpMid": "2", "sdpMLineIndex": 2, "candidate": "candidate:1576112651 1 udp 41885439 13.200.173.17 51014 typ relay raddr 13.127.175.223 rport 60447 generation 0 ufrag ks+P network-id 1"}

  1. Refer to Trickle ICE section to check if ICE candidates of type relay are being generated on the Tokkio UI side.

  2. Check if VST receives SDP offer from UE service.

[2025-4-11T5:35:33.192Z:128268684494400:PeerConnectionManager.cpp:2892: SendRemotePeerOffer]        [UEDBG] Sending offer from UE: {
    "apiKey" : "api/v1/streambridge/stream/start",
    "data" :
    {
            "sensorId" : "e9ce1949-5b3a-40d6-ab1b-508ae95aaea5",
            "sessionDescription" :
            {
                    "sdp" : "v=0\r\no=- 5002745187336916864 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS pixelstreaming_audio_stream_id pixelstreaming_video_stream_id\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:PgMU\r\na=ice-pwd:FlBARQTUxDvq1nceyw+AXFK1\r\na=ice-options:trickle\r\na=fingerprint:sha-256 3A:B8:51:72:74:93:EE:C8:92:34:14:76:16:B8:84:2C:19:62:61:72:C9:FB:29:66:96:F1:B9:32:93:22:3B:B9\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 urn:3gpp:video-orientation\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=sendonly\r\na=msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP9/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 red/90000\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 ulpfec/90000\r\na=ssrc-group:FID 1854727606 1930387950\r\na=ssrc:1854727606 cname:C1W15Ew52VpmMJ6A\r\na=ssrc:1854727606 msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label\r\na=ssrc:1854727606 mslabel:pixelstreaming_video_stream_id\r\na=ssrc:1854727606 label:pixelstreaming_video_track_label\r\na=ssrc:1930387950 cname:C1W15Ew52VpmMJ6A\r\na=ssrc:1930387950 msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label\r\na=ssrc:1930387950 mslabel:pixelstreaming_video_stream_id\r\na=ssrc:1930387950 label:pixelstreaming_video_track_label\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 63 110\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:PgMU\r\na=ice-pwd:FlBARQTUxDvq1nceyw+AXFK1\r\na=ice-options:trickle\r\na=fingerprint:sha-256 3A:B8:51:72:74:93:EE:C8:92:34:14:76:16:B8:84:2C:19:62:61:72:C9:FB:29:66:96:F1:B9:32:93:22:3B:B9\r\na=setup:actpass\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=sendrecv\r\na=msid:pixelstreaming_audio_stream_id pixelstreaming_audio_track_label\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 maxaveragebitrate=510000;maxplaybackrate=48000;minptime=3;sprop-stereo=1;stereo=1;usedtx=0;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:110 telephone-event/48000\r\na=maxptime:120\r\na=ptime:20\r\na=ssrc:2639110141 cname:C1W15Ew52VpmMJ6A\r\na=ssrc:2639110141 msid:pixelstreaming_audio_stream_id pixelstreaming_audio_track_label\r\na=ssrc:2639110141 mslabel:pixelstreaming_audio_stream_id\r\na=ssrc:2639110141 label:pixelstreaming_audio_track_label\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:PgMU\r\na=ice-pwd:FlBARQTUxDvq1nceyw+AXFK1\r\na=ice-options:trickle\r\na=fingerprint:sha-256 3A:B8:51:72:74:93:EE:C8:92:34:14:76:16:B8:84:2C:19:62:61:72:C9:FB:29:66:96:F1:B9:32:93:22:3B:B9\r\na=setup:actpass\r\na=mid:2\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n",
                    "type" : "offer"
            },
            "streamId" : "e9ce1949-5b3a-40d6-ab1b-508ae95aaea5_1"
    },
    "peerId" : "e9ce1949-5b3a-40d6-ab1b-508ae95aaea5_1"
}

Both streams are connected, but data is not received in the inbound stream (Avatar).#

If both streams are connected but a black screen is observed, that means data for the Avatar stream may not be received by the Tokkio UI. If both streams are connected, then VST is out of the picture since now there is peer-to-peer communication between the Tokkio UI and the UE service. To debug this issue, check the UE service logs and WebRTC stats in the web browser by going to the URL chrome://webrtc-internals/.

Resolution#

The most common solution is to restart the A2F service, the UE service, and check if it fixes the issue. If this doesn’t work, try restarting the SDR of the UE service and A2F as well.

On this page