FAQ

This section provides categorized frequently asked questions.

Blueprint & Agent Deployment

Are VIOS and VST the same service?

VIOS (Video IO & Storage) is also referred to as VST (Video Storage Toolkit). The Warehouse documentation uses VIOS for the service and UI; URLs such as /vst/ and configuration variables such as VST_USE_SDRC retain the VST name.

How do I resolve a failure when deploying the blueprints and agents?

We recommend the following steps to resolve deployment failures -

• Make sure a previous deployment isn’t already running using docker ps. If a previous deployment exists, then stop the deployment and run the app again.
• A failure can happen if the previous deployment was not stopped and a re-deployment was performed.
• Make sure the correct values are set in $VSS_APPS_DIR/industry-profiles/warehouse-operations/.env and overrides.env.
• Make sure the deployment user can read and write the directory configured by VSS_DATA_DIR.
• If you intentionally need a clean redeployment, follow Teardown the Deployment to remove the deployment volumes and data.
• Use docker compose ps to identify unhealthy services, then inspect a service with docker compose logs <service>. We recommend starting with the DeepStream and Kafka services.
• A port being occupied is a common cause for deployment failure.

Why are my configuration changes not reflected after redeployment?

The Blueprint Configurator reapplies only the settings it manages during deployment. For a managed setting, change its source value in the Warehouse .env, overrides.env, or blueprint_config.yml file. For an unmanaged setting, edit the target configuration file directly after cleanup.

For a clean redeployment, stop the deployment, run cleanup from $VSS_APPS_DIR, and only then edit the configuration and redeploy:

bash scripts/cleanup_all_datalog.sh \
  -e industry-profiles/warehouse-operations/overrides.env

Why does vss-elasticsearch-init fail during deployment (race condition with Elasticsearch)?

Observation: When deploying the blueprint with docker compose up, the elasticsearch-init-container service (vss-elasticsearch-init container) starts and tries to connect to Elasticsearch before it is fully ready. The init container uses all retry attempts (default 20) and exits with code 1, while Elasticsearch becomes healthy soon after.

Example log output from the vss-elasticsearch-init container:

Attempting to connect to the Elasticsearch server for ILM policy creation.
Unable to connect to ES. Trying to reconnect - (attempt 1/20)
...
Unable to connect to ES. Trying to reconnect - (attempt 20/20)
Max attempts to connect to ES reached.
Exiting Script.

Container status after failure:

vss-elasticsearch-init Exited (1)
elasticsearch Up (healthy)

This can also be caused by network issues on the host. Try increasing ELASTICSEARCH_CONNECTION_MAX_ATTEMPTS in $VSS_APPS_DIR/industry-profiles/warehouse-operations/.env, then redeploy.

Why does Kibana show request timeouts or ELK services fail startup health checks?

If Kibana shows Request timeout: "Request timed out", inspect the relevant service logs before changing Kibana settings.

A common root cause is Elasticsearch disk pressure. In Kibana Discover, this can appear as Unable to retrieve search results with Request timeout: "Request timed out". Check Elasticsearch allocation and disk watermarks before tuning Kibana timeouts. See the Elasticsearch disk watermark FAQ below.

If Docker marks Elasticsearch, Kibana, or Logstash as unhealthy during startup, that is a separate Docker health check timeout. Check the service logs first; if the service is only starting slowly, increase the affected service’s healthcheck.retries value in video-search-and-summarization/deploy/docker/services/infra/compose.yml (or deploy/docker/services/infra/compose.yml from the deployment repository root) and redeploy.

Why is CPU usage high?

For Docker Compose, use docker stats to identify which services are consuming the most CPU. For Helm deployments, use kubectl top pods -n <namespace>; this command requires the Kubernetes Metrics API. Then review the configuration and logs for the affected services or pods.

For example, Kafka-based deployments run four parallel Logstash analytics pipelines (mdx-kafka, mdx-kafka-1, mdx-kafka-2, and mdx-kafka-3), in addition to the separate mdx-lvs pipeline. This parallelism can increase CPU usage on hosts with fewer CPU cores.

For Docker Compose deployments, reduce the analytics pipeline count in $VSS_APPS_DIR/services/infra/elk/logstash/configs/pipelines-kafka.yml. Comment out extra pipeline.id entries and their corresponding path.config lines, keeping at least mdx-kafka, then restart Logstash. Fewer pipelines can reduce CPU overhead but also reduce ingestion throughput. Leave mdx-lvs unchanged. Redis deployments already use one analytics pipeline.

Another example is high CPU usage from PyTorch thread oversubscription when running Sparse4D. See PyTorch thread oversubscription (Sparse4D) under Performance Notes for tuning guidance.

Why does Elasticsearch stop accepting writes or mark indices read-only when disk usage is high?

Elasticsearch uses disk watermarks to protect the ELK stack from running out of disk space. One common symptom is Kibana Discover failing with Unable to retrieve search results and Request timeout: "Request timed out". The default watermarks are:

• Low watermark (default 85% disk used): Elasticsearch avoids allocating new shards to the node.
• High watermark (default 90% disk used): Elasticsearch starts relocating shards away from the node when possible.
• Flood stage (default 95% disk used): Elasticsearch sets affected indices to read_only_allow_delete so writes stop before the disk fills completely.

Check disk allocation and the active watermark settings:

curl --location "http://<ip>:9200/_cat/allocation?v"
curl --location "http://<ip>:9200/_cluster/settings?include_defaults=true&filter_path=**.disk.watermark*"

If flood stage is reached, fix the disk pressure first. Common options are:

• Remove unnecessary local data or logs from the Elasticsearch host.
• Remove unnecessary containers which are not being used.
• Add disk capacity or add Elasticsearch capacity so shards have somewhere safe to move.

After disk usage drops below the high watermark, Elasticsearch normally removes the block automatically. If an index remains read-only after disk pressure is fixed, clear it manually:

curl --location --request PUT "http://<ip>:9200/*/_settings?expand_wildcards=all" \
  --header "Content-Type: application/json" \
  --data "{\"index.blocks.read_only_allow_delete\": null}"

Why am I not seeing data on the Kibana/2D Blueprint UI?

We recommend the following steps to resolve the issue -

• If you are using nvstreamer, verify that the video is present in the mount or that the files were uploaded correctly.
• Check if the VST is able to play all the streams. Alternatively, you could also check if the streams are playable using ffplay, VLC or testRTSPClient.
• Check the logs of DeepStream to ensure the FPS is showing up correctly for the input streams.
• Inspect Kibana to see if the data is being sent. If it’s not, check the Logstash logs for any errors. If no errors are found, write a Kafka consumer script to verify if raw data is being sent by DeepStream.
• If the UI is loading partially, ensure that you have added all the pre-requisite files, such as the calibration file, images, and image metadata json file. Ensure that the image’s name matches with the value present in image metadata json file.
• Finally, make sure that the sensor IDs listed in the calibration file matches with the ones mentioned in DeepStream configuration. Also ensure that the sensorIDs of the streams that get added directly via VST matches with the ones mentioned in calibration file.

My videos produce incorrect timestamps for messages or are processed over an extended duration

• For DeepStream to properly work with NVStreamer the input videos need to be encoded to not contain any b frames. To check if the b-frames are present in a video, use either of the following commands:

mediainfo <media_file> | grep bframes
ffprobe <media_file> -show_frames | grep -E 'pict_type=B'

• You can encode your videos to this format by running

ffmpeg -i input_video.mp4 -vcodec libx264 -crf 25 -x264opts "bframes=0:keyint=30" encoded_output_video.mp4

Update the value of the keyint with the fps of your video.

I get the message “unauthorized: authentication required” when downloading containers from NGC

Follow How to get an NVIDIA NGC API key, then sign in to the container registry:

docker login nvcr.io --username '$oauthtoken'

Enter your NGC API key when prompted for the password, then retry the download.

Troubleshooting NVStreamer/VST Videos in Docker-Compose

• Verify that the NVStreamer/VST is serving the videos by opening the RTSP streams in either vlc or ffplay. If using VLC click on “Media” and “Open Network Stream” and enter the following RTSP URL - rtsp://<video url>. Stream RTSP URL can be grabbed from NVStreamer/VST UI.
• If NVStreamer/VST is not streaming the videos make sure the port 8554(NVStreamer)/8555(VST) is not blocked by a fire-wall.

How can I view VIOS video and overlays without a TURN server?

If WebRTC playback cannot connect because a TURN server is unavailable, select DASH instead of WebRTC from Streaming Protocol in the VIOS UI. DASH supports live and recorded video with overlays and does not require a TURN server. A TURN server is still required for WebRTC when the browser cannot connect directly to VIOS.

When I run reference application in playback mode could I loop through the playback data more than once?

The number of passes is controlled by the playbackLoop setting in the behavior analytics configuration. To replay the data immediately without redeploying, restart vss-behavior-analytics-playback for the 2D and 3D profiles, or vss-behavior-analytics-playback-mv3dt for the MV3DT profile.

To configure the number of passes before deployment, edit the existing playbackLoop entry under app in:

$VSS_APPS_DIR/industry-profiles/warehouse-operations/warehouse-<mode>-app/vss-behavior-analytics/configs/vss-behavior-analytics-config.json

Replace <mode> with 2d, 3d, or mv3dt. For example, the following value loops through the data 10 times:

{ "name": "playbackLoop", "value": "10" }

What if I don’t want to expose Kafka service when deploying it on different CSPs using docker compose?

Edit the kafka service in $VSS_APPS_DIR/services/infra/compose.yml. Bind the published port to the loopback interface and advertise the loopback address:

environment:
  KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:${KAFKA_HOST_PORT:-9092}
ports:
  - 127.0.0.1:${KAFKA_HOST_PORT:-9092}:9092

Containers in the Compose network continue to use kafka:29092, while the host port is reachable only from the deployment host.

What are effective tripwire counts?

A person may loiter along the tripwire and this will result in multiple IN & OUT events based on the tripwire definition which can be x number of detections before and after the tripwire. The web-API returns two counts: effective count and actual count. The UI displays the effective count which takes care of the loitering condition. See metrics-tripwire-counts in the Video Analytics API documentation for more information.

Why is the deepstream showing low FPS for certain sensors?

This issue may originate with the input sensors itself. Go to RTSP Stream QOS section in Debug tab of VST UI to check if the sensors are streaming with correct FPS. FPS details can be checked by enabling the Stream QOS stats and then selecting the particular sensor.

If the input sensors do not have any issues, then it could be caused due to overutilization of GPU.

For docker compose environment, current utilization of the GPU can be checked using nvidia-smi in CLI.

If you are consistently observing low FPS, try reducing the number of streams assigned to each GPU.

For additional tuning guidance, see Performance Notes in the Object Detection and Tracking documentation.

Why does vss-rtvi-cv crash with NvBufSurfTransform error -2 when adding streams on IGX Thor Mini?

On IGX Thor Mini, vss-rtvi-cv can log the following error: NvVicCompose Failed NvBufSurfTransform failed with error -2 while converting buffer. After the restart, vss-rtvi-cv may remain idle with Active sources : 0 while the SDR controller exhausts retries to add streams.

Check the kernel logs for VIC power-up failures:

sudo dmesg | grep -i vic

If the logs show tegra-vic job submission failures because the VIC engine could not be powered up, the board may have a VIC issue. As a workaround (WAR), set scaling-compute-hw: 1 in industry-profiles/warehouse-operations/warehouse-2d-app/deepstream/configs/ds-pgie-config.yml and redeploy the blueprint.

This change forces the PGIE scaling operation to run on the GPU, which may lower FPS.

Agents

Why does incident report generation time out when LLM reasoning is enabled?

Incident retrieval and video analysis can complete while incident report generation still fails. The agent returns no report artifacts and a message is returned “Report Agent: Failed to generate incident report: Failed to generate incident report”. A workaround for this is to disable LLM reasoning. In the Warehouse chat sidebar, open Agent Parameters, turn off LLM Reasoning, and request the report again.

Why is incident report generation missing the report body?

When LLM reasoning is enabled and the LLM max_tokens budget is used up, the report generated can be missing sections. As a workaround, increase the LLM max_tokens or turn reasoning off llm_reasoning: false in the profile config file.

VST

This section provides answers to some of the frequently asked questions on using VST application:

Why do streams fail to restart after removing and re-adding them in VST?

When using NVStreamer synchronized playback, all streams specified by nv_streamer_sync_file_count must request playback within nv_streamer_sync_data_arrival_timeout_secs (default: 150 seconds). Re-adding streams too slowly or adding fewer than the configured count can prevent playback.

Increase nv_streamer_sync_data_arrival_timeout_secs for slower re-adds. A longer timeout does not help if you re-add fewer streams than nv_streamer_sync_file_count, since playback waits for the full count. If you still cannot re-add every stream in time, or the stream count has changed, update the deployment configuration and redeploy the Warehouse Blueprint.

Why are object IDs missing from the video overlay?

In VST’s Analytics Overlay Settings, enable Show Bounding Boxes and Show Object ID under Object ID Display for the 2D and 2D with Agents profiles, then click Save Settings. Debug Mode is optional for these profiles and adds diagnostic overlays. For 3D and MV3DT, object IDs are displayed by default with Debug Mode enabled. See Display Object IDs in Warehouse Profiles for the settings and screenshot.

I bought a new camera, plugged it in, used default password but cannot access video

Some cameras require that username/password is modified prior to the first use as a part of a third-party system (like - for example - VST). In such situations, you will need to connect to the web page hosted by the camera itself. Attach your camera to the network bypassing VST host machine. Take note of its IP address, and launch a web page hosted by the camera (usually https://192.168.1.xxx). The web page will prompt you to alter the default username/password combination. Use your preferred settings.

I configured NVStreamer and VST but I don’t see any streams in VST

Please check if the videos are uploaded to NVStreamer, and also check if the volume mounted in NVStreamer is correct and has the videos. Also check if the RTSP URLs or NVStreamer endpoint is configured correctly in VST.

I started Video Wall with n streams, but I see less than n streams in the Video Wall

Please check if the streams are showing No Error in the VST UI Dashboard.

My camera was always streaming OK before. After I plugged it to the VST host through a switch (or directly) the VST web UI tells me it found my camera, but cannot stream from it.

The most common reason for VST reporting inability to stream is a mismatch between camera’s credentials, and default credentials stored by VST. For example - if you used the camera in the past with your preferred username/password credentials - you will need to input those credentials again through VST’s web UI. To do so - open the web UI, navigate to Device Management - Set Credentials, and update VST accordingly.

When I try to play live video stream I see a constantly rotating white circle

We suggest Chrome on Windows, macOS or Linux platforms. We are aware of potential video streaming issues when using alternative browsers like Firefox or Safari.

When I try to play recorded video stream I don’t see any video

Please check if recording is enabled for the camera.

I purchased a camera that is not on the list of tested ONVIF-S devices

All ONVIF-S compliant devices should work with this VST release. However, we noticed that not all cameras claiming ONVIF-S compatibility are fully compliant. If you have already exhausted your debugging options, contact us with the model and manufacturer of the camera you would like added to the compatibility list.

I see 0 FPS after restarting the perception container

If perception is restarted, you may observe 0 FPS. SDR adds streams to perception only once, during application startup. To restore FPS and stream processing, restart the entire blueprint instead of restarting only the perception container. Follow the Quickstart to redeploy the blueprint.

Handling Time Drift in ONVIF Cameras

This section provides answers to frequently asked questions on handling time drift issues in ONVIF cameras:

Why does time drift occur in ONVIF cameras?

Time drift typically arises due to the following reasons:

• The camera’s internal clock lacks high precision.
• Irregular or absent synchronization with a Network Time Protocol (NTP) server.
• Network latency or instability.
• Power outages or firmware-related issues.

How does time drift affect downstream systems like VIOS, Perception, and Analytics?

When these systems rely on frame timestamps provided by the camera, time drift can lead to:

• Incorrect timestamps on recorded video.
• Challenges correlating frames across multiple cameras.
• Difficulties during incident investigation or forensic analysis.

What is the recommended solution?

We recommend a dual-pronged approach to mitigate time drift:

1. Disable Sensor NTP Time in VIOS:

   • By default, the configuration parameter use_sensor_ntp_time is set to true. In this mode, VIOS uses the SenderReport and RTP-timestamp provided by the camera and forwards the same timestamp to downstream services such as Perception.
   • However, if time drift is observed between the camera clock and the system clock, set use_sensor_ntp_time to false:

   {
       "data": {
           "use_sensor_ntp_time": false
       }
   }
   • When this option is disabled, VIOS ignores the SenderReport and RTP-timestamp from the camera and assigns its own system timestamp when forwarding frames to downstream services like Perception. This ensures consistent and reliable timestamps based on the system clock, helping mitigate issues caused by camera time drift.

2. Enable NTP Synchronization on the Camera:

   • Configure each ONVIF camera to synchronize with a reliable NTP server (e.g., pool.ntp.org or a local enterprise NTP server).
   • Set synchronization to occur at regular intervals (e.g., every 1 hour) to minimize drift.
   • Ensure the camera supports NTP settings either via its web interface or ONVIF Device Management API.

How do I configure NTP on an ONVIF camera?

Most ONVIF cameras allow NTP setup through:

• Web Interface: Typically under Admin Settings > Time Settings > NTP Configuration
• ONVIF Device Management API: For programmatic configuration via VMS or automation tools

Ensure the following:

• The NTP server address is reachable by the camera.
• Time zone and Daylight Saving Time (DST) settings are correct.

Is NTP synchronization alone sufficient?

No. While NTP synchronization helps reduce time drift, it might not be precise or frequent enough for time-sensitive applications. Disabling use_sensor_ntp_time remains the preferred and more reliable approach for maintaining accurate timing.

What if the camera does not support NTP or cannot sync correctly?

If a camera lacks proper NTP support or experiences syncing issues:

• Set use_sensor_ntp_time to false so that VIOS applies the system timestamp to each frame when received.
• If feasible, replace the camera with one that fully supports NTP and ONVIF time configuration.

What if the cameras are already well-synchronized and show no drift?

In such cases, developers can choose to enable a specific VIOS configuration:

"use_sensor_ntp_time": true

This setting instructs VIOS to propagate the original timestamp from the camera to downstream services like Perception, which then uses it to assign time to each frame.

Model Update

Can I reuse a TensorRT engine file on another GPU?

I finetuned my own Sparse4D model via the Nvidia TAO Toolkit. How do I replace the existing model in the perception microservice with my own?

Refer to the 3D-multi-camera-detection-and-tracking-Sparse4D page for more details.

How do I modify the Sparse4D model to detect/track a different set of classes?

Model Finetuning via TAO Toolkit will be required to modify the model to detect & track a different set of classes. Refer to the Sparse4D model page for more details.

Install Fabric Manager

For H100 SXM HBM3 to run local LLM, Fabric Manager is required.

1. Download the Fabric Manager Debian package:

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/nvidia-fabricmanager_595.58.03-1ubuntu1_amd64.deb

2. Install the downloaded package:

   sudo dpkg -i nvidia-fabricmanager_595.58.03-1ubuntu1_amd64.deb

3. Start the Fabric Manager using the following command:

   sudo systemctl enable nvidia-fabricmanager
   sudo systemctl start nvidia-fabricmanager

4. Verify the Fabric Manager is running using the following command:

   sudo systemctl status nvidia-fabricmanager

5. Verify the Fabric Manager version using the following command:

   nv-fabricmanager -v

SSH Access

Here are the key assumptions and requirements for setting up passwordless SSH access:

1. SSH Key Pair Requirements:

   • A public/private key pair must be generated on the client machine
   • The public key must be added to the ~/.ssh/authorized_keys file on the target server
   • The private key must be kept secure on the client machine

2. File Permissions:

   • ~/.ssh directory should have 700 permissions (drwx------)
   • ~/.ssh/authorized_keys should have 600 permissions (-rw-------)
   • Private key files should have 600 permissions (-rw-------)

3. SSH Configuration:

   • SSH daemon must be running on the target server
   • SSH configuration should allow public key authentication
   • Default port is 22 (can be changed in /etc/ssh/sshd_config)

4. Common Issues to Check:

   • SELinux/AppArmor restrictions
   • Firewall rules blocking SSH
   • SSH daemon configuration (/etc/ssh/sshd_config)
   • User home directory permissions

5. Basic Setup Steps:

# On client machine
ssh-keygen -t rsa -b 4096
ssh-copy-id user@remote-server
# Or manually copy public key
cat ~/.ssh/id_rsa.pub | ssh user@remote-server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

6. Testing:

ssh -v user@remote-server
#The `-v` flag provides verbose output to help diagnose any issues.

7. Security Considerations:

   • Use strong key types (RSA 4096, Ed25519)
   • Consider using a passphrase for the private key
   • Regularly rotate keys
   • Use ssh-agent for key management
   • Consider restricting key usage with command= or from= options in authorized_keys

8. Troubleshooting:

   • Check SSH daemon logs: /var/log/auth.log or journalctl -u sshd
   • Verify key permissions
   • Test with verbose logging
   • Check SELinux/AppArmor logs if applicable

These are the fundamental assumptions and requirements for setting up passwordless SSH access.

How to get an NVIDIA NGC API key

1. Create an NGC Account

• Sign up at ngc.nvidia.com if you don’t already have an account
• Complete the registration process

2. Generate Your API Key

• Log in and open NGC API Keys, or select Account Settings from your account menu and find API Keys.
• Select Generate Personal Key, enter a name and expiration, and select the services needed to access your containers, models, and resources.
• Generate the key and store it securely; it is shown only once. See the NGC User Guide for details.

3. API Key Types Available

Personal API Key (Recommended)

• Tied to your user lifecycle within the NGC organization
• Configurable permissions and services access
• Configurable time-to-live (1 hour to “never expires”)
• Multiple keys can be created and managed

Legacy API Key

• Original NGC API key type
• Only one key at a time (generating new key revokes previous)
• Not supported by the Cosmos3 Reasoner NIM (1.7.0+). Use a Personal API Key for local VLM deployments.

4. Using Your NGC API Key

ngc config set

Enter your API key when prompted.

With Docker:

docker login nvcr.io
Username: $oauthtoken
Password: <Your_API_Key>

Important: For Docker login, always use $oauthtoken exactly as the username - this is a special authentication token for all users.

IGX Thor

nvidia-cdi-refresh.service does not start on boot

Symptom: After ISO installation, NVIDIA containers fail to run.

docker: Error response from daemon: CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=1

This can occur because nvidia-cdi-refresh.service did not start correctly on boot.

Fix: Enable and start the service:

sudo systemctl enable --now nvidia-cdi-refresh.service