SIL Architecture

System architecture for SIL testing environment.

System Diagram

Components

1. Isaac Sim

Isaac Sim provides the simulation environment for SIL testing: 20×20m warehouse with loading dock, trailer, racking, and pallet storage. Materials and lighting match real-world conditions for camera feeds.

Forklift: Spawned from the fleet file (robots.yaml spawn:) into a generated layer over the scene. Swivel-wheel kinematics — back_wheel_drive (velocity), back_wheel_swivel (steering). Safety indicator disc at /World/forklift_b/body/body/safety_indicator changes color by safety state (the mute decision, or the proximity level under PSF_APP=pxc). Physics handles steering, friction, weight distribution, and collision detection.

Digital humans: IRA (Isaac Replicator Agent) for autonomous navigation. Path planning, obstacle avoidance, and animation blending. Agent count and behavior configurable in default_config_ros.yaml.

Cameras: 3 cameras (overhead, side, loading dock). 1920×1080 RGB. Isaac Sim 6.1 self-hosts RTSP in-process — one server per camera on ports 8554/8555/8556. With --enable-vst, cameras auto-register via cameras.yaml.

Cameras stream at 15 FPS in both 2D and 3D mode. 3D additionally requires a unique timestamp per frame for Sparse4D temporal fusion, which the system (wall-clock) timestamps provide.

ROS2: The control graph subscribes to /forklift_b/cmd_vel and the odometry graph publishes /forklift_b/odom (names from the fleet file); the indicator graph subscribes to /safety/is_muted from the Communication Layer, or to /forklift_b/proximity/state from the forklift-controller under pxc; /clock carries sim time. Uses jazzy, rmw_cyclonedds_cpp, domain ID 0.

2. VSS Warehouse Blueprint

Perception pipeline consuming camera streams and generating safety events.

2D profile: DeepStream with RT-DETR model for object detection and NvDCF for per-camera tracking. Publishes events to Kafka mdx-events topic.

3D profile: DeepStream with Sparse4D model for 3D multi-view BEV detection and MTMC (Multi-Target Multi-Camera) tracking. Detections and tracks go to the Kafka mdx-bev topic; behavior analytics reads them and publishes the ROI, tripwire and proximity events to mdx-events, which is the seam the Safety Core reads in both profiles. See 3D Vision AI Profile and Sparse4D documentation for details.

The pipeline monitors ROI and Tripwire zones, publishing events to Kafka.

Event format:

{
  "event": {
    "type": "EXIT",
    "id": "roi-id-1",
    "info": {
      "class": "roi"
    }
  },
  "object": {
    "type": "Forklift",
    "id": "125",
    "confidence": 0.826,
    "speed": 15.409
  },
  "sensor": {
    "id": "Camera_01"
  },
  "timestamp": "2026-01-26T20:27:20.366Z"
}

Event types:

3. Safety Core (PSF)

Safety event processing:

Safety Event Integrator (SEI):

• Consumes Kafka events on mdx-events for ROI and tripwire alerts and mdx-frames for per-frame analysis, in both the 2D and the 3D profile
• 2D (nvpss.conf, deploymentMode = 2D): multi-camera fusion with timeWindowSize=6000 ms (the shipped SIL value); ATL events bypass fusion
• 3D (nvpss_bev.conf, deploymentMode = 3D): fusion off — perception already fused the views, and events arrive under one BEV sensor, bev-sensor-1, declared as a group of the three cameras
• Event validation
• Forwards validated events to Decision Maker

Monitors: the Safety AI Monitor (PSF_LAUNCH_MODE) and the Perception Monitor (PSF_PCM_MODE) both ship skip, and both nvpss files declare them not required (required_safety_monitor = 0, required_perception_monitor = 0, required_mdx_instances = 1). The container refuses to start when a key does not match its mode; see Required monitors.

Safety Decision Maker (SDM):

PSF_APP picks the decision maker; each one reads its event binding from configs/<app>_sdm.conf.

• atl (default) — Automated Trailer Loading: tracks forklift and human presence in the ROI and generates MUTE (0x02) or UNMUTE (0x07), sent as UDP packets (64 bytes) to the Communication Layer on port 12346. With two bays (atl-dual-override.yaml) zone 2 sends to COMM_UDP_PORT_ZONE2 (12347).
• pxc — Proximity: scores Forklift × Person separation (EVENT_12/13/14) and generates NORMAL (0x07), REDUCE (0x05) or STOP (0x02).
• both — the two in one container (--app atl,pxc); proximity sends to COMM_UDP_PORT_PXC (12348).

4. Communication Layer

Protocol bridge between Safety Core and Isaac Sim:

• UDP Receiver (port 12346, plus 12347 / 12348 when zone 2 / proximity send on their own): Receives Safety Core commands
• OPC UA Server (port 4840): Industrial protocol interface; zone 2 has its own nodes (Zone2IsMuted), and a proximity safe-release request is a write of True to Safety.ProximitySafeReleaseRequest
• ROS2 Bridge: Publishes the mute to /safety/is_muted (Bool) and to /<robot>/safety/is_muted for each robot in COMM_ROBOT_IDS, and proximity decisions to /safety/proximity/pair (JSON) and /safety/proximity/mode. Proximity never touches /safety/is_muted: its opcodes 0x02 and 0x07 mean the opposite of ATL’s

ROS2 configuration:

• ROS_DISTRO: jazzy
• RMW_IMPLEMENTATION: rmw_cyclonedds_cpp
• ROS_DOMAIN_ID: 0

Logs commands to: ${MDX_DATA_DIR}/comm-layer/opc_server.log

Data Flow

2D mode:

Isaac Sim Cameras (3 cams, 15 FPS)
     ↓ RTSP streams
AI Perception (VSS Warehouse Blueprint)
     ├─ RT-DETR detection
     ├─ NvDCF tracking
     └─ ROI/Tripwire analytics
     ↓ Kafka events (mdx-events topic)
Safety Core (PSF)
     ├─ SEI: Multi-camera fusion + event validation
     └─ SDM: ATL decision logic (MUTE/UNMUTE)
     ↓ UDP packet (64 bytes, port 12346)
Communication Layer
     ├─ OPC UA update
     └─ ROS2 publish (/safety/is_muted)
     ↓ ROS2 topic
Isaac Sim Action Graph
     └─ Update safety indicator (GREEN/ORANGE)

3D mode:

Isaac Sim Cameras (3 cams, 15 FPS)
     ↓ RTSP streams (unique timestamps)
AI Perception (VSS Warehouse Blueprint)
     ├─ Sparse4D 3D multi-view BEV detection + tracking
     └─ BEV ROI/Tripwire/proximity analytics
     ↓ Kafka events (mdx-events topic, sensor id = bev-sensor-1)
Safety Core (PSF, deploymentMode = 3D)
     ├─ SEI: Event validation + passthrough (fusion done at perception)
     └─ SDM: ATL (MUTE/UNMUTE) and/or proximity (NORMAL/REDUCE/STOP), per PSF_APP
     ↓ UDP packets (ATL port 12346; proximity 12348 under both)
Communication Layer
     ├─ OPC UA update
     └─ ROS2 publish (/safety/is_muted; /safety/proximity/pair)
     ↓ ROS2 topics
Isaac Sim Action Graph + forklift-controller
     ├─ Update safety indicator (mute, or proximity level)
     └─ Slow / stop the forklift on REDUCE / STOP (pxc, both)

Differences from Production

SIL runs on x86 with Docker Compose. Key differences from production (IGX Thor):

Shared between SIL and production: AI Perception (VSS Warehouse Blueprint) pipeline, Safety Core fusion algorithms, ATL decision logic, and event formats — ensuring SIL validation reflects production behavior.

Next Steps

• Isaac Sim Configuration - Isaac Sim configuration details
• Isaac Sim Action Graphs - Forklift ROS2 Action Graph
• Forklift Controller - The service that drives the forklift along a route
• SIL — 2D Perception - Deploy SIL system (2D)
• SIL — 3D Perception - Deploy SIL system (3D)