Known Issues#

  • Audio Processing and CV pipeline do not work together for now. You can try either of the two at a time.

  • Models are trained on specific data/use cases so if tested on other inputs then it might give incorrect results.

  • VLM Model accuracy: Sometimes time stamps returned are not accurate. Also, it can hallucinate for certain questions. Prompt tuning might be required.

  • Summarization accuracy: Summarization accuracy is heavily dependent on VLM accuracy. Also, the default configs have been tuned for the warehouse use case. User can supply custom VLM and summarization prompts to the /summarize API.

  • The following harmless warnings might be seen during VSS application execution. These can be safely ignored.

    • GLib (gthread-posix.c): Unexpected error from C library during ‘pthread_setspecific’: Invalid argument. Aborting

    • (gst-plugin-scanner:1883): GStreamer-WARNING : 08:52:11.807: Failed to load plugin …

    • gradio.exceptions.Error: ‘Chunk Size must be a valid number’

  • Due to a browser limitation, loading multiple Gradio sessions in the same browser can cause Gradio sessions to get stuck or appear to be slow.

  • Guardrails might not reject some prompts that are expected to be rejected. This could be because the prompt might be relevant in other contexts and topics in the prompt might not be configured to be rejected. You can try tuning the guardrails configuration if required.

  • OpenAI connection errors or 429 (too many requests) errors might be seen sometimes if too many requests are sent to GPT-4v or GPT-4o VLMs. It can be due to lower TPM/RPM limits associated with the OpenAI account.

  • CA-RAG Summarization might show a truncated summary response. This is due to the max_tokens. Try increasing the number in the CA-RAG config file.

  • Helm deployment: VSS deployment pod fails with Error: (LLM call Exception: llm-nim-svc)

    Despite having an init container wait for LLM pod to come up, VSS deployment can for an unknown reason error out like below.

    2024-11-27 17:51:44,763 [91mERROR[0m Failed to load VIA stream handler - LLM Call Exception: HTTPConnectionPool(host='llm-nim-svc', port=8000): Max retries exceeded with url: /v1/chat/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2c9d0ad6c0>: Failed to establish a new connection: [Errno 111] Connection refused'))

    If this happens, wait for additional few minutes and a pod restart fixes the issue.

    Users can monitor this using sudo watch microk8s kubectl get pod.

  • CV pipeline is currently supported for video files and live-streams only. Images are not supported.

  • Deleting RTSP streams can be hung sometimes. This is because, rtspsrc indefinitely retries TCP transport after UDP timeout when the timeout property is set. After TCP link is established, pipeline teardown hangs. This is a GStreamer issue: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1570. Workaround can be exporting VSS_RTSP_TIMEOUT=0. This will disable TCP transport after UDP timeout. But, this could cause streaming to not work at all when network is not good.

  • Some video encoded formats such as H.264 High 4:4:4 Profile (Hi444pp) are not supported. For such videos summarization request will fail and “No summary was generated for given request” message will be shown. In VSS container logs you will observe an error similar to below:

    Error String : Feature not supported on this GPUError Code : 801

  • If you observe “An internal error occurred” as the output, the likely cause is that the Initialization failed. This is because the VSS container failed to start. Check the logs for more details.

    One of common reasons is that VSS is not able to connect to the embeddings or reranking service.

  • In “IMAGE FILE SUMMARIZATION & Q&A”, When one of the samples is selected, VSS produce a summary with multiple timestamps, even though only one image is selected.

    The VLM, CA RAG prompts by default are for video/multi image.

    More info about setting these prompts in the UI: Image Summarization, Q&A, and Alerts.

    The users need to prompt tune for single image to avoid this issue.

    More info about tuning these prompts: Tuning Prompts.

  • Rarely Guardrails failure might cause VSS to stop. In such cases, either Guardrails can be disabled or a custom VSS container can be built with following changes:

    diff --git a/src/vss-engine/src/via_stream_handler.py b/src/vss-engine/src/via_stream_handler.py
index e4fb893..6fed288 100644
--- a/src/vss-engine/src/via_stream_handler.py
+++ b/src/vss-engine/src/via_stream_handler.py
@@ -1265,7 +1265,8 @@ class ViaStreamHandler:
                             )
                         except Exception as e:
                             logger.error("Error in guardrails: %s", str(e))
-                            self.stop(True)
+                            with self._lock:
+                                self._LLMRailsPool.append(rails)
                             raise Exception("Guardrails failed")
                         # Return the rails to the pool
                         with self._lock:
@@ -1506,7 +1507,8 @@ class ViaStreamHandler:
                     response = rails.generate(messages=[{"role": "user", "content": query}])
                 except Exception as e:
                     logger.error("Error in guardrails: %s", str(e))
-                    self.stop(True)
+                    with self._lock:
+                        self._LLMRailsPool.append(rails)
                     raise Exception("Guardrails failed")
                 # Return the rails to the pool
                 with self._lock:
@@ -1904,7 +1906,8 @@ class ViaStreamHandler:
                     response = rails.generate(messages=[{"role": "user", "content": query}])
                 except Exception as e:
                     logger.error("Error in guardrails: %s", str(e))
-                    self.stop(True)
+                    with self._lock:
+                        self._LLMRailsPool.append(rails)
                     raise Exception("Guardrails failed")
                 # Return the rails to the pool
                 with self._lock:

  • Rarely, issuing a high number of file summarization requests back to back for long period of time might cause VSS to stop.

  • Sometimes highlight generation can fail if the LLM is not able to generate a valid JSON response. You can retry the request.

  • In some cases, Cosmos-Reason1 model may require to enable reasoning using enable_reasoning in the API to extract the IDs overlayed by the CV pipeline. So VSS will automatically force reasoning when CV metadata is enabled. This behaviour can be disabled setting the environment variable VSS_FORCE_CR1_REASONING_FOR_CV_METADATA=false. However, even if reasoning is enabled, some object IDs may still not be extracted.

  • Some VSS responses may contain HTML / markdown tags which can cause the VSS UI output to be rendered incorrectly.

  • Configuring alerts results in incorrect summaries or alerts do not get generated on x86 platforms and Jetson Thor. This has been fixed in VSS GitHub repository. Need to build the VSS Engine container from source to fix the issue.

  • Using a smaller LLM model like Llama 3.1 8B for RAG may result in lower accuracy. Try using a larger model like Llama 3.1 70B