Skip to navigation
Closed-Loop Testing

SIL — 2D Perception

View as Markdown

Deploy Halos SIL for Software-in-Loop testing.

Halos SIL requires VSS Warehouse 3.3 as its perception backend. Deploy VSS Warehouse first, as in Deploy AI Perception, with the SIL overrides of Step 2, before proceeding with Halos SIL deployment.

Prerequisites

See Prerequisites for hardware and software requirements.

For an automated bring-up with a coding agent (Claude Code, Codex, Cursor, and similar), use the hoisa-deploy-profile skill — it replaces Steps 1–6 below with a single agent prompt. See Agentic Deployment.

Manual instructions below remain supported and are required if you are not using a coding agent.

Step 1: Download Packages

Clone the Halos Outside-In Safety repository (ships the compose files, profiles, configs, and scripts in-tree), then download the SIL data tarball from NGC:

# Clone the Halos Outside-In Safety repo
git clone https://github.com/NVIDIA/halos-outside-in-safety.git -b v1.4.0
cd halos-outside-in-safety
# Download SIL data
ngc registry resource download-version \
  "nvidia/halos-outside-in/sample-sil-data:v1.4.0"
tar -xvf sample-sil-data_v*/halos-outside-in-sil-data.tar.gz \
  --directory=/home/$USER/ # change me

Output:

  • halos-outside-in-safety/ - Cloned repo (compose files, profiles, configs, scripts)
  • sil-data/ - Isaac Sim scenes and configs

Step 2: Deploy VSS Warehouse

Deploy VSS Warehouse 3.3.0, the GitHub release v3.3.0 of the VSS repository, as in Deploy AI Perception, or with the VSS repository’s vss-build-vision-ai skill (warehouse profile). Apply the overrides below before docker compose up — if perception builds its TensorRT engine with the wrong config, it has to build it again.

Before the first up:

  • Log Docker in to nvcr.io with your NGC API key. The release images, nvcr.io/nvidia/vss-core/*:3.3.0, do not pull anonymously; without the login a pull stops at unauthorized: authentication required.

    echo "$NGC_CLI_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

  • Download the app data, nvidia/vss-warehouse/vss-warehouse-app-data:3.3.0, one 2.1 GB archive. Without --org nvidia --team vss-warehouse the download stops at Missing org - If Authenticated, org is also required.

    ngc registry resource download-version \
      "nvidia/vss-warehouse/vss-warehouse-app-data:3.3.0" --org nvidia --team vss-warehouse
    tar -xzf vss-warehouse-app-data_v3.3.0/vss-warehouse-app-data.tar.gz \
      --directory=/path/to # change me

    VSS_DATA_DIR is the extracted vss-warehouse-app-data/ directory. It carries no models: perception downloads them into <vss_data>/models/ at its first start.

  • Prepare the data directories on a fresh extract. The containers run as different users, so their data directories must exist and be writable by all of them; otherwise Redis restarts in a loop on Can't open the log file: Permission denied. This is the data-directory step of the vss-build-vision-ai skill:

    DATA=/path/to/vss-warehouse-app-data   # VSS_DATA_DIR
    for d in analytics_cache calibration_toolkit elastic/data elastic/logs kafka redis/data redis/log \
             vss_video_analytics_api vst/clip_storage nvstreamer/vst_data; do
      mkdir -p "$DATA/data_log/$d"
    done
    mkdir -p "$DATA/agent_eval/dataset" "$DATA/agent_eval/results" "$DATA/models"
    chmod -R a+rwx "$DATA/data_log" "$DATA/agent_eval" "$DATA/models" "$DATA/videos"

<wh_ops> below is the VSS warehouse-operations directory, <vss_repo>/deploy/docker/industry-profiles/warehouse-operations/. Edit the override files there in place, but do not run docker compose from <wh_ops>/ — its compose.yml is only an app fragment and pulls in no infrastructure (Kafka, Redis, sdr-controller). Deploy from <vss_repo>/deploy/docker/ with containers.env, industry-profiles/warehouse-operations/.env and industry-profiles/warehouse-operations/overrides.env, in that order.

2.1 overrides.env Overrides

Set these in <wh_ops>/overrides.env, which Docker Compose reads last:

HOST_IP='<HOST_IP>'                                             # get with: hostname -I | awk '{print $1}'
COMPOSE_PROFILES=${COMPOSE_PROFILES_WH_KAFKA_2D_MINIMAL}        # default: …_WH_KAFKA_2D, which adds Elasticsearch and monitoring
SAMPLE_VIDEO_DATASET="warehouse-loading-dock-3cams-synthetic"   # default: warehouse-4cams-20mx20m-synthetic
NUM_STREAMS=3                                                   # default: 4; matches 3 Isaac Sim cameras
HARDWARE_PROFILE=L40S                                           # default: H100; the perception GPU
LLM_NAME_SLUG=none                                              # default: nemotron-3.5-lightning-30b-a3b; no LLM for SIL
VSS_APPS_DIR="<vss_repo>/deploy/docker"
VSS_DATA_DIR="/path/to/vss-warehouse-app-data"

MODE=2d, BP_PROFILE=bp_wh_kafka, STREAM_TYPE=kafka, LLM_MODE=none and VLM_MODE=none already ship; keep them. The Safety Core reads its events from Kafka: with bp_wh, or a profile list without Kafka, it receives none while every container still looks healthy.

Export NGC_CLI_API_KEY in the shell that runs Docker Compose: perception downloads its models with it at start. containers.env pins the images to nvcr.io/nvidia/vss-core, tag 3.3.0 (VSS_CONTAINER_TAG); leave it.

2.2 DeepStream Config

File: <wh_ops>/warehouse-2d-app/deepstream/configs/ds-main-config.txt

Disable SEI extraction in [source-list] (the app config ships it enabled):

[source-list]
# extract-sei-type5-data=1
# sei-uuid=NVDS_CUSTOMMETA

Set the timing keys in [streammux]:

[streammux]
attach-sys-ts-as-ntp=1       # timestamp each frame with host (wall-clock) arrival time
# extract-sei-sim-time=1     # SEI off — do not use Isaac SEI sim-time as the timestamp
# drop-backward-sei=1        # moot with SEI extraction off

Isaac Sim 6.1 RTSP does embed SEI, but the Safety Core does not consume sim time — using it as the frame timestamp makes the Safety Core drop every event as STALE. System timestamps keep decisions flowing. Without these changes the perception pipeline also reports low FPS and bounding boxes flicker on VST.

2.3 Perception Model Config

Nothing to change. 2D runs the RT-DETR model that warehouse-2d-app downloads into <vss_data>/models/ at start, and the shipped warehouse-loading-dock-3cams-synthetic calibration is used as-is. (3D downloads its Sparse4D model the same way; see SIL — 3D Perception.)

2.4 VST Config

File: <wh_ops>/warehouse-2d-app/vst/configs/vst_config.json

Several copies of vst_config.json exist in the deploy tree. Edit the one the VST container actually mounts.

"rtsp_streaming_over_tcp": true,
"use_sensor_ntp_time": false,
"bbox_tolerance_ms": 100
  • rtsp_streaming_over_tcp: true — ingest Isaac’s self-hosted RTSP over TCP. UDP ingest can stall RTP delivery silently (see Isaac Sim Streaming).
  • use_sensor_ntp_time: false — pair with attach-sys-ts-as-ntp=1 above so VST and DeepStream agree on wall-clock arrival time.
  • bbox_tolerance_ms: 100 — widen the metadata-to-frame match window (default 0) to reduce bbox flicker on VST.

This file also carries a halo_safety_* block. Leave halo_safety_udp_port at its shipped -1 for the sil profile — SIL drives the safety indicator through the comm-layer / ROS 2 path, not the VST overlay. Only the base profile sets it to 12345.

2.5 Wait for the TensorRT Engine

Start VSS Warehouse. The first startup downloads the model and builds the TensorRT engine, which takes several minutes. Poll vss-rtvi-cv until all 3 cameras report a non-zero FPS:

docker logs vss-rtvi-cv 2>&1 | grep -E 'Active sources|PERF' -A3 | tail -6

Do not use docker logs -f here — it follows the stream and never returns.

Expected output when ready:

Active sources : 3
**PERF:
15.00000 (14.99911)     source_id : 2 stream_name Camera_02
15.00000 (14.99919)     source_id : 1 stream_name Camera_01
15.00000 (15.00077)     source_id : 0 stream_name Camera

All 3 sources must report non-zero FPS. A source stuck at 0.00000 isn’t arriving — Isaac Sim is not streaming yet, the RTSP URL is wrong, or the engine is still building.

Then confirm the Safety Core event seam has data:

docker exec kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic mdx-events --max-messages 1 --timeout-ms 30000

Do not proceed to Halos SIL until both signals fire: 3 cameras at FPS > 0, and mdx-events has data.

Step 3: Configure Halos SIL Environment

nano deployments/profiles/sil.env

Required settings:

MDX_SAMPLE_APPS_DIR="/path/to/halos-outside-in-safety"  # this repo checkout root
MDX_DATA_DIR="/path/to/sil-data"                        # the extracted sil-data dir from Step 1
HOST_IP='<HOST_IP>'                                      # same value as the VSS overrides.env
ISAAC_GPU_DEVICE=1                                       # an RT-core GPU that does not run perception
ROS_DOMAIN_ID=0                                          # MUST be unique per host (0-232); see Troubleshooting
DOCKER_GID=999                                           # run: getent group docker | cut -d: -f3

PSF_IMAGE (the PSF 1.4 Safety Core image, nvcr.io/nvidia/halos-outside-in/outside-in-safety:nv-psf-halos-1.4-6b92731) and ISAAC_SIM_IMAGE (nvcr.io/nvidia/isaac-sim:6.1.0) are pre-set — leave them. The 1.4 configurations do not run on the 1.3 image.

PSF_LAUNCH_MODE (Safety AI Monitor) and PSF_PCM_MODE (Perception Monitor) ship skip. To run one active, also set its key to 1 in closed-loop-testing/safety-core/configs/nvpss.conf (required_safety_monitor or required_perception_monitor). Otherwise the Safety Core stops at start with Error: --saim-mode active conflicts with required_safety_monitor = 0 in /opt/nvidia/psf/bin/nvpss.conf. See Required monitors.

Two more settings choose the run, and their defaults are the 2D test scenario:

  • SCENARIO=warehouse_20x20_1fl names the scene, fleet file, cameras config and waypoints together (closed-loop-testing/isaac-sim/sil/configs/scenarios.yaml); Isaac and every controller read it.
  • PSF_APP (unset = atl) picks the decision maker. pxc and both (proximity) expect the 3D feed; see SIL — 3D Perception and Proximity Monitoring in the Deployment Guide.

Step 4: Deploy Halos SIL

cd deployments
../closed-loop-testing/scripts/setup.sh sil              # Create required directories and set permissions
../closed-loop-testing/scripts/cleanup_all_datalog.sh sil # Clean previous data logs
docker compose --env-file profiles/sil.env up -d --build

Services started: safety-core, comm-layer, isaac-sim, forklift-controller.

Step 5: Run Test Scenario

docker exec -d isaac-sim bash -lc 'cd /isaac-sim/sil/scripts && \
  ./run_sdg.sh --start --headless --enable-vst'

SCENARIO from sil.env supplies the IRA config, the fleet file and the cameras config. -c, --robots-config and --cameras-config still win over it for a one-off run.

For script usage details and additional options, see Execution Script.

What happens:

  1. Loads warehouse scene (20×20m)
  2. Spawns the forklift from the fleet file (robots.yaml) and 3 digital humans
  3. Initializes ROS2 Action Graph for forklift
  4. Runs forklift playback sequence
  5. Starts RTSP streaming — Isaac 6.1 self-hosts one RTSP server per camera (8554/camera, 8555/camera_01, 8556/camera_02). See RTSP Multi-Camera Graph for the graph that builds them.
  6. Registers 3 cameras with VST (--enable-vst deletes existing sensors, then adds the Isaac cameras once the render is warm)

Startup time:

  • First run: 10–15 min — Isaac Sim loads its cache and compiles render shaders, then starts the simulation and brings up RTSP streaming.
  • Subsequent runs: 5–10 min — cache and shaders are already compiled.
  • Simulation: runs until you stop it. --start streams continuously rather than for a fixed frame count.

Step 6: Monitor Safety Commands

The log paths below use MDX_DATA_DIR from your sil.env. From the deployments/ directory, load it into your shell first (re-run it in each terminal you monitor from):

set -a; source profiles/sil.env; set +a

Monitor OPC server log:

tail -f $MDX_DATA_DIR/comm-layer/opc_server.log

Expected output:

INFO:udp_receiver.safety_receiver:Received: Seq#0 | HEARTBEAT | 💓 Heartbeat | ts=2026-04-29T13:33:28.055302+00:00
INFO:udp_receiver.safety_receiver:Received: Seq#3 | MUTE (ALLOW OPERATION) | 🟢 Safety muted - Loading allowed | ts=2026-04-29T13:33:38.946800+00:00
INFO:udp_receiver.safety_receiver:Received: Seq#9 | UNMUTE (PREVENT OPERATION) | 🟡 Safety active + Alarm on | ts=2026-04-29T13:34:06.987138+00:00

🟢 MUTE (ALLOW OPERATION): Forklift in trailer, no humans detected (allow loading)

🟡 UNMUTE (PREVENT OPERATION): Human detected or forklift exiting (alarm active)

💓 HEARTBEAT: Periodic keep-alive (every 5s) — confirms PSF→comm-layer link is healthy

Monitor PSF:

tail -f $MDX_DATA_DIR/psf-log/pss.log

Expected output:

2026-04-29T13:33:38.770555+00:00 host nv_mdx_client[59]:  Timestamp: 2026-04-29 13:33:38:770421 Endpoint: NVPSB_PSS_SOURCE Data: Safety event reported: EVENT_0 (rule: Forklift tripwire OUT)
2026-04-29T13:34:06.823078+00:00 host nv_mdx_client[59]:  Timestamp: 2026-04-29 13:34:06:822968 Endpoint: NVPSB_PSS_SOURCE Data: Safety event reported: EVENT_1 (rule: Forklift tripwire IN)
2026-04-29T13:34:06.986286+00:00 host NVPSB_PSD_CLIENT[34]:  Timestamp: 2026-04-29 13:34:06:986216 Endpoint: NVPSB_PSD_CLIENT  Data: PSD-Gateway: received DecisionRequest id=1 with 1 events

EVENT_0 / EVENT_1: Tripwire crossings reported by perception (forklift OUT/IN trailer).

DecisionRequest: PSF decision-maker invoked — produces the corresponding MUTE/UNMUTE command on the OPC log above.

Step 7: View Camera Streams

VST UI — http://<HOST_IP>:30888/vst/

# Open in browser
http://<HOST_IP>:30888/vst/

View live camera streams from Isaac Sim.

Camera stream view in VST

Cleanup and Restart

Restart the Scenario (Live Stack)

--start runs the scenario until it is stopped. To relaunch the Isaac scenario on an already-running stack, use the wrapper — it pauses VST stream processing for the boot window (so the fresh RTSP mounts don’t hit the cold-start “no caps” race), then resumes it once the streams are live:

bash closed-loop-testing/scripts/restart_isaac.sh   # reuses the running driver's own args

Expect the mounts to warm in ~1–2 min (cached shaders), VST to resume, and DeepStream back at 3/3 cameras ~30 s later.

Reset to a Fresh Deployment

When a stack misbehaves in ways a restart doesn’t fix, return it to the clean first-bring-up state without paying the TensorRT/shader rebuild. Stop the scenario and the Halos stack, tear VSS down with down -v (it drops the VST sensor database volume; the models and the TensorRT engine live under VSS_DATA_DIR/models/ and are kept), redeploy VSS, then clear Redis before relaunching:

docker exec redis redis-cli FLUSHALL
docker restart sdr-controller
docker restart vss-rtvi-cv

Keep VSS_DATA_DIR/models/ and the isaac-cache/ shader dirs — rebuilding them costs several minutes each.

Full Cleanup

# Stop Halos SIL
cd deployments
docker compose --env-file profiles/sil.env down
bash ../closed-loop-testing/scripts/cleanup_all_datalog.sh sil
docker volume prune -f
# Stop VSS Warehouse: see "Stop VSS Warehouse Blueprint" in
# the Deploy AI Perception page (same env files as the deployment)

docker volume prune -f removes every dangling volume on the host, not only this stack’s. With the VSS stack down that includes the VST sensor database. Skip it unless you want that reset, or remove the specific volumes instead.

Troubleshooting

Safety Core (PSF) Cannot Connect to Kafka

Warehouse (Kafka) must be running before Halos SIL. Check status:

# Check Warehouse running
docker ps | grep kafka

If Kafka is not running, redeploy VSS Warehouse following Step 2: Deploy VSS Warehouse.

STALE Events in Safety Core (PSF)

First check that DeepStream stamps frames with system time (2.2). nvpss.conf ships timeWindowSize = 6000 (ms) for SIL; raise it only if events still arrive STALE:

nano closed-loop-testing/safety-core/configs/nvpss.conf
# Increase: timeWindowSize
cd deployments && docker compose --env-file profiles/sil.env restart safety-core

Safety Indicator Flickering (Multi-Machine)

If multiple SIL systems run on the same network, each must use a unique ROS_DOMAIN_ID (0-232) in deployments/profiles/sil.env. Without this, ROS2 nodes from different machines publish to the same /safety/is_muted topic, causing cross-machine interference and safety indicator flickering.

# In deployments/profiles/sil.env — assign a unique number per machine
ROS_DOMAIN_ID=42

Restart Halos SIL after changing. Verify with:

docker exec comm-layer bash -c \
  "source /opt/ros/jazzy/setup.bash && ros2 topic info /safety/is_muted -v"

Should show Publisher count: 1.

Known Issues

Upstream Isaac Sim and VST issues affect the stream path; their status on Isaac Sim 6.1 and the workarounds are in Isaac Sim Streaming. They are not specific to 2D.

Next Steps