Deploy AI Perception

This quick start guide covers setting up the environment and deploying the Halos Outside-In Safety Blueprint with sample data.

NVIDIA Halos Outside-In Blueprint Architecture

The current architecture consists of two main groups of components. This page uses the user-facing names. The Integration Guide describes the same system as architectural sub-elements, and the mapping is direct: AI Perception is the Sensor Input Processing Pipeline (SIPP), and Safety Core covers the Safety AI Monitor (SAIM), Perception Container Monitor (PCM), Safety Event Integrator (SEI), and Safety Decision Maker (SDM), along with the Safety Black Box (SBB) and Safety User Interface (SUI) that sit outside the mainstream data path.

This page covers AI Perception; the Safety Core components are introduced in Deploy the Safety Core.

AI Perception (Sensor Input Processing Pipeline)

This module detects safety events from 2D camera streams using advanced perception and behavior analytics tuned for warehouse environments. It continuously tracks people, forklifts, AMRs, and other moving assets and emits semantically rich events tied to safety zones and workflows. The system supports three primary event types:

• Region-of-interest (ROI) events: Triggered when objects enter or exit configured safety zones, enabling occupancy monitoring and enforcement of human‑free or equipment‑free areas.
• Tripwire events: Triggered when objects cross virtual lines placed at doors, dock edges, or aisle boundaries, enabling directional counting and presence inference in partially occluded or blind areas.
• Proximity events: Triggered when people and equipment come within unsafe distances, supporting pedestrian–vehicle collision avoidance and automatic slow‑down or stop behaviors.

Learn more about AI Perception component: VSS Warehouse Blueprint - 2D Vision AI and Perception Microservices Documentation.

Deploying Halos Outside-In

Halos Outside-In can be deployed in three levels:

The deployment process includes these steps:

1. Camera Selection, Placement and Calibration
2. Installation of VSS Warehouse Blueprint
3. Installation of Safety Event Integrator and Decision Maker

Camera Selection, Placement and Calibration Guidelines

(If you just want to try out Halos Outside-In with the sample videos packaged within the software, skip this step and proceed to the next step Installation of AI Perception.)

Camera model choice and configuration settings significantly impact detection latency. The method used to connect cameras to the compute platform greatly influences overall system latency.

Best Practices for Deployment

• Use a Consistent Camera Model: Deploy the same camera model across all units to ensure uniform performance and predictable latency.

• Standardize Camera Settings: Apply identical configuration settings to every camera in the deployment. This consistency is essential for reliable detection and optimal fusion of results by the safety supervisor, minimizing latency across the system.

• Connection Method Matters: Ensure that all cameras connect to the compute platform using the same interface and topology to avoid introducing variable delays.

Optimal camera placement is crucial and should be tailored to your specific use case. Camera positioning significantly affects both accuracy and latency of safety event detection. For detailed guidance on camera positioning for best results, refer to the camera placement guide.

Once you place your camera, calibrate it by referring to the camera calibration guidelines.

Configuring Safety Zones and Tripwires is part of the camera calibration process and definitions of tripwires and zones are included in the calibration file.

Installation of AI Perception

System Requirements

The System and GPU requirement of AI Perception (VSS Warehouse Blueprint - 2D profile) is shown as below:

The stream counts are the 2D ceilings in the VSS blueprint configurator for each HARDWARE_PROFILE. The sample dataset used below has 3 cameras, which fits every row.

The minimum system requirements are:

• x86-64 architecture
• 18 core CPU
• 128 GB RAM
• 1 TB SSD
• 1 × 1 Gbps network interface
• 1 or 2 supported GPUs

For more details in hardware and software prerequisites please refer to VSS Warehouse Blueprint Prerequisites.

Installation Steps

1. Clone the VSS Warehouse repository

Halos Outside-In 1.4 runs on VSS Warehouse 3.3.0, GitHub release v3.3.0. Its container images are the release images in nvcr.io/nvidia/vss-core, tag 3.3.0; you need an NGC API key to pull them, for the sample data and for the models perception downloads at start. If you don’t have the NGC CLI installed, refer to the NGC CLI installation guide: https://org.ngc.nvidia.com/setup

git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout v3.3.0

The warehouse-operations profile, including the sample calibration data, lives under deploy/docker/industry-profiles/warehouse-operations/.

2. Download the sample data assets

Download the sample app data (videos and playback for the warehouse-loading-dock-3cams-synthetic dataset) from NGC and extract it:

ngc registry resource download-version \
  "nvidia/vss-warehouse/vss-warehouse-app-data:3.3.0" \
  --org nvidia --team vss-warehouse
tar -xzf vss-warehouse-app-data_v3.3.0/vss-warehouse-app-data.tar.gz -C /path/to

Then prepare the data directories, the data-directory step of the VSS repository’s vss-build-vision-ai skill:

DATA=/path/to/vss-warehouse-app-data   # VSS_DATA_DIR
for d in analytics_cache calibration_toolkit elastic/data elastic/logs kafka redis/data redis/log \
         vss_video_analytics_api vst/clip_storage nvstreamer/vst_data; do
  mkdir -p "$DATA/data_log/$d"
done
mkdir -p "$DATA/agent_eval/dataset" "$DATA/agent_eval/results" "$DATA/models"
chmod -R a+rwx "$DATA/data_log" "$DATA/agent_eval" "$DATA/models" "$DATA/videos"

• Without --org nvidia --team vss-warehouse, the NGC CLI stops at Missing org - If Authenticated, org is also required.
• The archive extracts with mode 750, and the containers run as different non-root users: without this step Kafka exits with cluster_id: Permission denied and Redis restarts in a loop on Can't open the log file: Permission denied. A directory still missing at up is created by Docker as root, which those users cannot write.
• The archive carries no models; perception downloads them into models/ at its first start.

3. Environment Settings

VSS 3.3 keeps the profile defaults in warehouse-operations/.env and the deployment settings in deploy/docker/industry-profiles/warehouse-operations/overrides.env, which Docker Compose reads last. Set these values in overrides.env; each comment gives the default where it differs:

MODE=2d
BP_PROFILE=bp_wh_kafka                                          # MUST include Kafka for the Safety Core
STREAM_TYPE=kafka
SAMPLE_VIDEO_DATASET="warehouse-loading-dock-3cams-synthetic"   # default: warehouse-4cams-20mx20m-synthetic
NUM_STREAMS=3                                                   # default: 4
# Match your GPU: run `nvidia-smi -L` to identify it. Valid values are in the table above.
HARDWARE_PROFILE=L40S                                           # default: H100
LLM_MODE=none
VLM_MODE=none
LLM_NAME_SLUG=none                                              # default: nemotron-3.5-lightning-30b-a3b
# Minimal service list: perception, behavior analytics, VST, Kafka and Redis.
COMPOSE_PROFILES=${COMPOSE_PROFILES_WH_KAFKA_2D_MINIMAL}         # default: the extended 2D Kafka list
VSS_APPS_DIR="/path/to/video-search-and-summarization/deploy/docker"
VSS_DATA_DIR="/path/to/vss-warehouse-app-data"                  # where the sample data was extracted in step 2
HOST_IP='<HOST IP>'

Export two values in the shell that runs Docker Compose. containers.env expands the image tag before any env file is read, and perception reads the NGC key from the environment; never write the key into a file:

export VSS_CONTAINER_TAG=3.3.0   # the release images in nvcr.io/nvidia/vss-core
export NGC_CLI_API_KEY=<your NGC API key>

4. Deploy VSS Warehouse Blueprint

echo "$NGC_CLI_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

for img in vss-rt-cv vss-behavior-analytics vss-vios-sensor vss-vios-streamprocessing vss-vios-ingress; do
  docker pull nvcr.io/nvidia/vss-core/$img:3.3.0
done

Run Docker Compose from deploy/docker/ so the relative include paths resolve. The top-level compose.yml pulls in the shared infrastructure (Kafka, Redis), the perception app, and the warehouse-operations profile. Pass the three env files in this order; overrides.env must come last to win:

cd deploy/docker
# Start the blueprint
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file industry-profiles/warehouse-operations/.env \
  --env-file industry-profiles/warehouse-operations/overrides.env \
  up --detach --force-recreate --build

The VSS repository’s vss-build-vision-ai agent skill (warehouse profile) produces the same deployment from the answers above; see Agentic Deployment.

Verify the running VSS Warehouse Blueprint:

# Verify if containers are in running state
docker ps
docker compose ls
# For 2D Vision AI Profile (perception)
docker logs -f vss-rtvi-cv
# Open a browser to check the data once all containers are up and running
# Verify the video storage toolkit (VST) dashboard
http://<HOST_IP>:30888/vst/

Before launching Safety Core, validate the streams and events that HOISA will consume:

# Confirm the VST sensors are online.
curl -s http://<HOST_IP>:30888/vst/api/v1/sensor/list
# Validate each exact rtspUrl configured in sensor_config.conf. Neither
# tool is installed by default: ffprobe comes from the ffmpeg package,
# gst-launch-1.0 from gstreamer1.0-tools with gstreamer1.0-plugins-good.
ffprobe -rtsp_transport tcp <rtsp_url>
# or
gst-launch-1.0 rtspsrc location=<rtsp_url> ! fakesink
# Optional: confirm behavior analytics events are present on mdx-events.
docker exec kafka kafka-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic mdx-events \
    --from-beginning \
    --timeout-ms 10000

Stop VSS Warehouse Blueprint:

# From deploy/docker, use the SAME env files you deployed with
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file industry-profiles/warehouse-operations/.env \
  --env-file industry-profiles/warehouse-operations/overrides.env \
  down -v --remove-orphans
# Clear the bind-mounted data (Kafka, Redis, VST under VSS_DATA_DIR/data_log).
# The containers' users own these files, so the cleanup needs root.
sudo env VSS_DATA_DIR=/path/to/vss-warehouse-app-data VSS_APPS_DIR=$PWD \
  bash scripts/cleanup_all_datalog.sh --env-file industry-profiles/warehouse-operations/.env

The models and the TensorRT engine stay under VSS_DATA_DIR/models/, so the next deployment reuses them.

Customization

It is supported to customize the VSS Warehouse Blueprint for user specific use cases, including adding new cameras, configuring different safety zones and tripwires, fine-tuning the AI perception models, etc.

For details please refer to the VSS Warehouse 2D Blueprint Customization Guide.

Installation of Safety Event Integrator and Decision Maker

Once the AI Perception is up and running, events are being generated from the input camera streams. To convert those events into actionable signals, continue with Deploy the Safety Core, which introduces the Safety Core components and walks the installation, launch, and verification steps.

Annex A: VSS Warehouse Blueprint deployment workarounds for safety-flashed IGX Thor

The two issues described in this annex have been observed only when deploying the VSS Warehouse Blueprint - 2D profile on safety-flashed IGX Thor systems (IGX Thor Developer Kit T7000 and Developer Kit Mini T5000). Deployment on other (non-safety-flashed) platforms is not affected.

Apply the corresponding workaround below before deploying on these systems. With these workarounds, VSS Warehouse 3.2.1 has been deployed successfully on safety-flashed IGX Thor systems. The IGX Thor flow has not been run on VSS 3.3 yet, so the commands in this annex are the VSS 3.2.1 ones (a single .env file).

a. nvstreamer-2d container fails to start (NVIDIA Container Runtime missing)

Issue: In the safety build with the Safety Extension Package (SEP), runtime: nvidia is commented out in warehouse-2d-app.yml and the NVIDIA Container Runtime is not selected automatically. As a result, the nvstreamer-2d container fails to start with a runtime hook error.

Workaround: Add runtime: nvidia to the nvstreamer-2d service via a Compose override. Create deploy/docker/docker-compose.override.yml with:

services:
  nvstreamer-2d:
    runtime: nvidia

The deployment command passes -f compose.yml explicitly, which disables auto-discovery of the override file, so add it with an extra -f flag when starting the blueprint:

cd deploy/docker
docker compose -f compose.yml -f docker-compose.override.yml \
  --env-file industry-profiles/warehouse-operations/.env \
  up --detach --force-recreate --build

With this override, the nvstreamer-2d service (container vss-vios-nvstreamer in docker ps) starts successfully.

b. Intermittent Docker image pull failures with —pull always

Issue: Deploying with --pull always re-fetches every service image on each docker compose up and intermittently fails, blocking the deployment during image pulls.

Workaround: Pre-pull the required images once while authenticated to nvcr.io (see the Docker registry authentication step under Installation of AI Perception), then start the blueprint without --pull always so Docker Compose uses the locally cached images:

cd deploy/docker
docker compose -f compose.yml \
  --env-file industry-profiles/warehouse-operations/.env \
  up --detach --force-recreate --build

Prefer a local cache or an authenticated registry over anonymous Docker Hub pulls.