SIL — 3D Perception

Deploy Halos SIL with 3D perception (Sparse4D multi-view fusion + BEV tracking).

The Halos services (safety-core, comm-layer, isaac-sim, forklift-controller) and the Isaac Sim launch are identical to 2D. 3D changes the VSS perception side — the app, its DeepStream timing config, and the behavior-analytics proximity settings — plus two Safety Core configuration lines in the profile: on a 3D feed the Safety Core runs deploymentMode = 3D for every PSF_APP.

2D vs 3D

Prerequisites

See Prerequisites for hardware and software requirements.

Additional 3D requirements:

• Familiarity with the VSS Warehouse Blueprint - 3D Vision AI Profile
• Understanding of Sparse4D 3D multi-camera detection

Step 1: Download Packages

Follow Step 1 in SIL — 2D Perception to clone the Halos Outside-In Safety repository and download the SIL data. The same repo and SIL data are used for both 2D and 3D.

3D needs no extra download. Perception downloads its Sparse4D model into <vss_data>/models/sparse4d/ before it builds the TensorRT engine: with DATASET_TYPE=synthetic that is RN50 v2.3 (nvidia/tao/sparse4d_rn50:deployable_v2.3), with real RN50 v3.0, both public in the nvidia org. The download uses NGC_CLI_API_KEY.

Step 2: 3D Calibration Dataset (Custom Scenes Only)

The loading-dock test scene needs nothing here. VSS 3.3 ships <wh_ops>/warehouse-3d-app/calibration/sample-data/warehouse-loading-dock-3cams-synthetic/ with the BEV group bev-sensor-1 and the top-level roi-id-1 / tripwire-id-1, under the same dataset name as the 2D one. Use it as the reference for the format, and follow this step only for a scene of your own.

A 3D calibration needs BEV sensor grouping and world-coordinate ROI / tripwire definitions. A scene that already has a 2D calibration converts structurally (adding fields), not by re-computation. Do this before the VSS deploy so 3D behavior analytics loads the ROIs / tripwires at startup. <name> is the dataset name, the same for both apps:

# <wh_ops> = <vss_repo>/deploy/docker/industry-profiles/warehouse-operations
Source: <wh_ops>/warehouse-2d-app/calibration/sample-data/<name>/calibration.json
Dest:   <wh_ops>/warehouse-3d-app/calibration/sample-data/<name>/calibration.json

Start from the 2D calibration.json and add the following.

2.1 Per Sensor — a BEV group

"group": {
  "name": "bev-sensor-1",
  "alias": "area-1",
  "type": "bev",
  "origin": [<from translationToGlobalCoordinates>],
  "dimensions": [<bounding box of all globalCoordinates points>]
}

group.name (bev-sensor-1) binds the three cameras into one BEV fusion group — every sensor in the group shares it.

2.2 Per Sensor — a region entry in place

{"name": "region", "value": "Region-1"}

2.3 Top-level rois and tripwires

ROIs and tripwires are defined in world coordinates, which are shared between 2D and 3D for the same physical scene — copy the roiCoordinates, wire, and direction values straight from the 2D calibration. Keep the ids matching nvpss.conf (roi-id-1, tripwire-id-1) and the object types matching the ATL event map (restrict Person, confine Forklift):

"rois": [
  {
    "id": "roi-id-1",
    "roiCoordinates": [
      {"x": 4.87747667, "y": -18.9756077},
      {"x": 4.87747667, "y": -11.2385153},
      {"x": 9.57440535, "y": -11.2385153},
      {"x": 9.57440535, "y": -18.9756077}
    ],
    "restrictedObjectTypes": ["Person"],
    "confinedObjectTypes": ["Forklift"],
    "sensors": ["Camera", "Camera_01", "Camera_02"],
    "groups": ["bev-sensor-1"]
  }
]

"tripwires": [
  {
    "id": "tripwire-id-1",
    "wire": {
      "p1": {"x": 9.57440535, "y": -18.9756077},
      "p2": {"x": 9.57440535, "y": -11.2385153}
    },
    "direction": {
      "p1": {"x": 8.0, "y": -15.1070615},
      "p2": {"x": 11.0, "y": -15.1070615}
    },
    "sensors": ["Camera", "Camera_01", "Camera_02"],
    "groups": ["bev-sensor-1"]
  }
]

2.4 Copy the BEV Floorplan Image

SRC=<wh_ops>/warehouse-2d-app/calibration/sample-data/<name>
DST=<wh_ops>/warehouse-3d-app/calibration/sample-data/<name>
mkdir -p "$DST/images"
cp "$SRC/images/Top.png"             "$DST/images/"
cp "$SRC/images/imageMetadata.json"  "$DST/images/"
file "$DST/images/Top.png"    # must report: PNG image data (a file, not a directory)

2.5 Verify

python3 -c "
import json
d = json.load(open('$DST/calibration.json'))
for s in d['sensors']:
    assert 'group' in s, f'{s[\"id\"]} missing group'
assert len(d.get('rois', [])) > 0, 'no ROIs defined'
assert len(d.get('tripwires', [])) > 0, 'no tripwires defined'
print(f'OK: {len(d[\"sensors\"])} sensors, {len(d[\"rois\"])} ROIs, {len(d[\"tripwires\"])} tripwires')
"

Then set SAMPLE_VIDEO_DATASET="<name>" in overrides.env (Step 3.1) and put the scene’s videos under <vss_data>/videos/<name>/. The calibration file must exist before up: for a missing calibration.json Docker creates a directory in its place and perception emits nothing.

Step 3: Deploy VSS Warehouse (3D Profile)

Deploy VSS Warehouse 3.3 as in Deploy AI Perception, or with the VSS repository’s vss-build-vision-ai skill (warehouse profile, 3D mode). The release, the nvcr.io login, the app data and the data directories are those of SIL — 2D Perception, Step 2. Apply the overrides below before docker compose up — if perception builds its TensorRT engine with the wrong config, it has to build it again.

3.1 overrides.env Overrides

Set the 2D values of SIL — 2D Perception (2.1) in <wh_ops>/overrides.env, with these two changed:

MODE=3d                                                         # selects warehouse-3d-app (Sparse4D + 3D behavior analytics)
COMPOSE_PROFILES=${COMPOSE_PROFILES_WH_KAFKA_3D_MINIMAL}

DATASET_TYPE=synthetic already ships and picks the Sparse4D bundle for synthetic scenes. SAMPLE_VIDEO_DATASET stays "warehouse-loading-dock-3cams-synthetic": the 3D app ships its BEV calibration under the same name.

3.2 DeepStream Config

File: <wh_ops>/warehouse-3d-app/deepstream/configs/ds-main-config.txt

Disable SEI extraction in [source-list] (the app config ships it enabled):

[source-list]
# extract-sei-type5-data=1    # SEI off for SIL — use system timestamps
# sei-uuid=NVDS_CUSTOMMETA     # comment out
low-latency-mode=0            # prioritise throughput over latency for 3D multi-view inference

Set the timing keys in [streammux]:

[streammux]
attach-sys-ts-as-ntp=1        # timestamp each frame with host (wall-clock) arrival time
# extract-sei-sim-time=1      # SEI off — do not use Isaac SEI sim-time as the timestamp
# drop-backward-sei=1         # moot with SEI extraction off
sync-inputs-ntp=0             # avoid NTP input-sync stalling on the Isaac RTSP feed
align-first-buffer=0          # do not gate the batch on first-buffer alignment
batched-push-timeout=75000    # looser batch timeout — Sparse4D needs longer to assemble a batch
drop-pipeline-eos=1           # one stream's EOS must not tear down the batched pipeline

3.3 Perception Model Config

File: <wh_ops>/warehouse-3d-app/deepstream/configs/config.yaml

num_sensors: 3           # must match NUM_STREAMS
num_torch_threads: 8     # CPU threads for the Torch post-process stage
gpu_postprocess: False   # run Sparse4D post-processing on CPU (avoids GPU contention)

The configurator sets num_sensors from NUM_STREAMS, and the other two keys ship with these values — nothing to edit here.

gpu_postprocess: False keeps Sparse4D post-processing on the CPU so it does not contend with inference on the GPU. If raising num_torch_threads does not buy you enough post-process throughput, set it to True to move that stage onto the GPU instead.

3.4 Behavior Analytics (pxc / both only)

File: <wh_ops>/warehouse-3d-app/vss-behavior-analytics/configs/vss-behavior-analytics-config.json

The shipped 3D config pairs only the humanoid robots with Person, within 2 m, so a forklift never produces a proximity event and the Halos tiers (REDUCE up to 3.5 m) would be cut off:

"name": "proximityDetectionThreshold",     "value": "4"
"name": "proximityDetectionCenterClasses", "value": "[\"Forklift\",\"Fourier_GR1_T2_Humanoid\",\"Agility_Digit_Humanoid\"]"

• proximityDetectionThreshold ships as 2 and must be at least every distance_threshold_meters in the proximity mapping (3.5 m): above it mdx-client drops the whole rule group, STOP included, without a log line.
• proximityDetectionCenterClasses ships without Forklift.

3.5 VST Config

File: <wh_ops>/warehouse-3d-app/vst/configs/vst_config.json

"rtsp_streaming_over_tcp": true,
"use_sensor_ntp_time": false,
"bbox_tolerance_ms": 100

• rtsp_streaming_over_tcp: true — ingest Isaac’s self-hosted RTSP over TCP. UDP ingest can stall RTP delivery silently (see Isaac Sim Streaming).
• use_sensor_ntp_time: false — pair with attach-sys-ts-as-ntp=1 so VST and DeepStream agree on wall-clock arrival time.
• bbox_tolerance_ms: 100 — widen the metadata-to-frame match window to reduce bbox flicker at the lower 3D frame rate.

3.6 Wait for the TensorRT Engine

Start VSS Warehouse and wait for the Sparse4D model download and TensorRT engine build (first startup: about 5 min on an L40S). Poll vss-rtvi-cv until all 3 cameras report a non-zero FPS:

docker logs vss-rtvi-cv 2>&1 | grep 'PERF' | tail -3

Expect ~15 FPS, the same as 2D. What matters is that all 3 sources report non-zero FPS; a source stuck at 0.00000 isn’t arriving.

Then confirm the Safety Core event seam has data (mdx-events, the same topic as 2D):

docker exec kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic mdx-events --max-messages 1 --timeout-ms 30000

Step 4: Configure Halos SIL Environment

nano deployments/profiles/sil.env

Required settings — the 2D ones, plus the two Safety Core 3D lines:

MDX_SAMPLE_APPS_DIR="/path/to/halos-outside-in-safety"  # this repo checkout root
MDX_DATA_DIR="/path/to/sil-data"                        # the extracted sil-data dir from Step 1
HOST_IP='<HOST_IP>'                                      # same value as the VSS overrides.env
ISAAC_GPU_DEVICE=1                                       # a GPU with RT cores + >20 GB, not running VSS perception
ROS_DOMAIN_ID=0                                          # MUST be unique per host (0-232); see Troubleshooting
DOCKER_GID=999                                           # run: getent group docker | cut -d: -f3
# 3D feed: required for every PSF_APP, atl included
PSF_NVPSS_CONFIG_SRC=./configs/nvpss_bev.conf
PSF_SENSOR_PIPELINES_CONFIG_SRC=./configs/sensor_pipelines_config_bev.conf

With the 2D nvpss.conf on a 3D feed, mdx-client drops every event (Dropping event: sensor not in sensor_config, so it has no pipelineID: bev-sensor-1) and no decision is made. With the two lines, the Safety Core runs deploymentMode = 3D with fusion off: the BEV app reports the three cameras as one sensor, bev-sensor-1, which configs/sensor_group_config.conf declares as group 4 of the cameras in sensor_config.conf.

PSF_IMAGE (the PSF 1.4 Safety Core image) and ISAAC_SIM_IMAGE are pre-set; the 1.3 image does not run these configurations. Both monitors ship skip, as in SIL — 2D Perception; to run one active, set its required_* key to 1 in nvpss_bev.conf.

4.1 Safety App (PSF_APP)

• pxc and both are SIL-only and run on the 3D feed, with the behavior-analytics change of Step 3.4. The proximity mapping closed-loop-testing/safety-core/configs/proximity_event_mapping.pb.txt scores Forklift × Person centre to centre: STOP ≤ 2 m, REDUCE ≤ 3.5 m, as EVENT_12/13/14 (EVENT_0..11 are ATL’s).

• both needs the override file and a port of its own for proximity. Set both next to PSF_APP in sil.env:

  PSF_APP=both
  COMPOSE_FILE=compose.yaml:../closed-loop-testing/safety-core/atl-pxc-override.yaml
  COMM_UDP_PORT_PXC=12348

  The Safety Core then runs the image’s --app atl,pxc: one gateway, one daemon, one mdx_client, and both decision makers, each on its own --sdm-config. ATL sends to COMM_UDP_PORT and proximity to COMM_UDP_PORT_PXC, and the comm-layer listens on both.

• The truck’s reaction is the robot’s proximity: block in robots.yaml (reduce_speed, stop_hold_s, reduce_hold_s, stale_s). A STOP holds stop_hold_s past the last packet asking for one; silence keeps the last level; stale_s (10 s) with no message at all stops the truck with fault psf_link_stale.

Step 5: Deploy Halos SIL

cd deployments
../closed-loop-testing/scripts/setup.sh sil              # Create required directories and set permissions
../closed-loop-testing/scripts/cleanup_all_datalog.sh sil # Clean previous data logs
docker compose --env-file profiles/sil.env up -d --build

Services started: safety-core, comm-layer, isaac-sim, forklift-controller — the same stack as 2D.

Step 6: Run Test Scenario

The Isaac Sim launch is identical to 2D:

docker exec -d isaac-sim bash -lc 'cd /isaac-sim/sil/scripts && \
  ./run_sdg.sh --start --headless --enable-vst'

For script usage details and additional options, see Execution Script.

What happens:

1. Loads the warehouse scene (20×20 m)
2. Spawns the forklift from the fleet file (robots.yaml) and 3 digital humans
3. Initializes the ROS2 Action Graph for the forklift
4. Runs the forklift waypoint sequence
5. Starts RTSP streaming — Isaac 6.1 self-hosts one RTSP server per camera (8554/camera, 8555/camera_01, 8556/camera_02). See RTSP Multi-Camera Graph for the graph that builds them.
6. Registers 3 cameras with VST once the render is warm (--enable-vst deletes existing sensors, then adds the Isaac cameras)

Step 7: Monitor Safety Commands

The log paths use MDX_DATA_DIR from your sil.env. From deployments/, load it into your shell first (re-run it in each terminal you monitor from):

set -a; source profiles/sil.env; set +a

Monitor the OPC server log:

tail -f $MDX_DATA_DIR/comm-layer/opc_server.log

Expected output:

INFO:udp_receiver.safety_receiver:Received: Seq#0 | HEARTBEAT | 💓 Heartbeat | ts=2026-04-29T13:33:28.055302+00:00
INFO:udp_receiver.safety_receiver:Received: Seq#3 | MUTE (ALLOW OPERATION) | 🟢 Safety muted - Loading allowed | ts=2026-04-29T13:33:38.946800+00:00
INFO:udp_receiver.safety_receiver:Received: Seq#9 | UNMUTE (PREVENT OPERATION) | 🟡 Safety active + Alarm on | ts=2026-04-29T13:34:06.987138+00:00

🟢 MUTE (ALLOW OPERATION): Forklift in trailer, no humans detected (allow loading)

🟡 UNMUTE (PREVENT OPERATION): Human detected or forklift exiting (alarm active)

💓 HEARTBEAT: Periodic keep-alive (every 5s) — confirms the PSF→comm-layer link is healthy

Monitor PSF:

tail -f $MDX_DATA_DIR/psf-log/pss.log

Expected output:

... nv_mdx_client[59]: ... Endpoint: NVPSB_PSS_SOURCE Data: Safety event reported: EVENT_0 (rule: Forklift tripwire OUT)
... nv_mdx_client[59]: ... Endpoint: NVPSB_PSS_SOURCE Data: Safety event reported: EVENT_1 (rule: Forklift tripwire IN)
... NVPSB_PSD_CLIENT[34]: ... Data: PSD-Gateway: received DecisionRequest id=1 with 1 events

EVENT_0 / EVENT_1: tripwire crossings reported by perception (forklift OUT / IN the trailer).

DecisionRequest: the PSF decision-maker is invoked — it produces the corresponding MUTE/UNMUTE command shown in the OPC log above.

Proximity (pxc / both): EVENT_12 (no violation), EVENT_13 (REDUCE tier) and EVENT_14 (STOP tier) appear in pss.log, the comm-layer logs each decision with the pair and its separation, and the forklift-controller logs every change of the level it applies:

INFO:udp_receiver.safety_receiver:Proximity: Seq#2591 | REDUCE (SAFE SPEED OPERATION) | machine[id=40 OBJECT] (7.83, -13.45, 0.00) | person[id=13 PERSON] (9.26, -11.66, 0.00) | sep=2.29m | ts=...

docker logs forklift-controller 2>&1 | grep "PXC \[" | tail -5
# PXC [forklift_b] normal -> reduce_speed | sep=3.33m | was 1.50 m/s, speed cap 0.50 m/s
# PXC [forklift_b] reduce_speed -> stop | sep=1.96m | was 0.50 m/s, speed -> 0

Step 8: View Camera Streams

VST UI — http://<HOST_IP>:30888/vst/

Navigate to Live Streams to view camera feeds with 3D detection overlays:

Navigate to Video Wall, enable overlay, and check Include Floor Plan in Analytics Overlay Settings to view the top-down BEV visualization with detected objects on the warehouse floorplan:

For details on configuring the Video Wall overlay and Floor Plan, see Playback overlay on Video Wall.

Cleanup and Restart

Restart the Scenario (Live Stack)

To relaunch the Isaac scenario on an already-running stack, use the wrapper — it pauses VST stream processing for the boot window (so the fresh RTSP mounts don’t hit the cold-start “no caps” race), then resumes it once the streams are live:

bash closed-loop-testing/scripts/restart_isaac.sh   # reuses the running driver's own args

Reset to a Fresh Deployment

When a stack misbehaves in ways a restart doesn’t fix, return it to the clean first-bring-up state without paying the TensorRT/shader rebuild. Stop the scenario and the Halos stack, tear VSS down with down -v (it drops the VST sensor database volume; the models and the TensorRT engine live under VSS_DATA_DIR/models/ and are kept), redeploy VSS, then clear Redis before relaunching:

docker exec redis redis-cli FLUSHALL
docker restart sdr-controller
docker restart vss-rtvi-cv

Keep VSS_DATA_DIR/models/ and the isaac-cache/ shader dirs — rebuilding them costs several minutes each.

Full Cleanup

# Stop Halos SIL
cd deployments
docker compose --env-file profiles/sil.env down
bash ../closed-loop-testing/scripts/cleanup_all_datalog.sh sil
docker volume prune -f
# Stop VSS Warehouse: see "Stop VSS Warehouse Blueprint" in
# the Deploy AI Perception page (same env files as the deployment)

Troubleshooting

Kafka mdx-events Topic Empty

The most common 3D issue. Verify sync-inputs-ntp=0 in the [streammux] section of ds-main-config.txt — with NTP input-sync on, the muxer stalls on the Isaac RTSP feed and produces zero Kafka output. Also confirm batched-push-timeout=75000 and low-latency-mode=0 (in [source-list]).

docker exec kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic mdx-events --max-messages 1 --timeout-ms 30000

DeepStream Stuck at ≤2/3 Active Sources

For 3D this is not healthy — Sparse4D fuses all 3 views (num_sensors: 3), so 2/3 corrupts the BEV. A camera stuck at 0 FPS after re-registration is usually a stale-identity zombie source bin (VST minted a new sensor uuid on restart; DeepStream kept the old one). Purge the stale source and add the live uuid — the procedure is in Recovering a camera stuck at 0 FPS.

No Decisions on the 3D Feed

pss.log full of Dropping event: sensor not in sensor_config ... bev-sensor-1 means the Safety Core still runs the 2D nvpss.conf. Set PSF_NVPSS_CONFIG_SRC and PSF_SENSOR_PIPELINES_CONFIG_SRC (Step 4) and recreate safety-core. This applies to every PSF_APP, atl included.

0 Detections

Check that perception downloaded its model (<vss_data>/models/sparse4d/) and that NGC_CLI_API_KEY was exported when VSS started: the download needs the key.

No Forklift × Person Proximity Events (pxc / both)

Behavior analytics still ships proximityDetectionThreshold 2 without Forklift as a center class. Apply Step 3.4 and recreate vss-behavior-analytics.

0 ROIs Loaded by Behavior Analytics

The calibration.json is missing the top-level rois array, or the container needs to be recreated (not just restarted) to pick up the newly bind-mounted calibration.

0 Behavior-Analytics Events (Frames Dropped as STALE)

The DeepStream SEI / system-timestamp override (Step 3.2) isn’t applied. With SEI extraction left on (or attach-sys-ts-as-ntp unset), PSF keys off Isaac’s sim-time and drops every synthetic frame as STALE, so mdx-behavior stays empty even though all containers are healthy. Set SEI off + attach-sys-ts-as-ntp=1 and recreate the affected containers. Since VSS 3.2.1 this is what replaces the retired inSimulationMode=true behavior-analytics flag.

Bounding Box Flickering on VST

Set bbox_tolerance_ms: 100 in vst_config.json. The default tolerance of 0 ms causes metadata-to-frame matching misses.

Safety Indicator Flickering (Multi-Machine)

If multiple SIL systems run on the same network, each must use a unique ROS_DOMAIN_ID (0-232) in deployments/profiles/sil.env. Verify with:

docker exec comm-layer bash -c \
  "source /opt/ros/jazzy/setup.bash && ros2 topic info /safety/is_muted -v"

Should show Publisher count: 1.

Safety Core (PSF) Cannot Connect to Kafka

VSS Warehouse (Kafka) must be running before Halos SIL:

docker ps | grep kafka

Known Issues

Upstream Isaac Sim and VST issues affect the stream path; their status on Isaac Sim 6.1 and the workarounds are in Isaac Sim Streaming. They are not specific to 3D.

Next Steps

• SIL Architecture — System architecture and data flow
• Isaac Sim Configuration — Isaac Sim configuration
• Isaac Sim Action Graphs — Forklift control details
• VSS Warehouse 3D Profile Customization — Adding cameras, custom scenes
• Sparse4D Configuration — Tuning 3D perception
• VSS Configurator — Sensor source configuration