Is this page helpful?

Known Limitations#

This section summarizes current behavioral constraints in the HOISA Safety Core.

ATL Use Case#

Single-forklift control model#

The Automated Trailer Loading (ATL) reference application is designed and validated for controlling a single forklift operating within the monitored loading dock and trailer region.
If multiple forklifts enter the same Region of Interest (ROI) or trailer area concurrently, the Safety Event Integrator and Decision Maker may not be able to reliably associate safety control commands with the intended forklift under control, which can disrupt safe operation.

Event Type Coverage#

Supported safety event types#

The HOISA Safety Core, as released, supports a limited set of safety events from the perception pipeline for decision making:
- Region of Interest (ROI) occupancy
- Tripwire crossings
- SafetyAreaViolation events generated by person presence in restricted zones
These events are sufficient to drive the ATL reference application, but they do not yet exploit the full richness of available perception metadata.

Isaac Sim Streaming#

Upstream RTSP and VST issues#

Four open issues affect the Isaac Sim → VST stream path on Isaac Sim 6.0. They are upstream (Isaac/Omniverse and VST/VIOS), not Halos Safety Core issues, and they apply to any profile that runs Isaac Sim — 2D and 3D alike. Apply the workaround and continue.

Bug ID	Symptom	Workaround
6477478	Isaac/Omniverse. RTSP over UDP: RTP delivery silently dies while the control plane stays healthy. Two modes — probe-triggered (~4 min) and spontaneous (after hours).	Ingest over TCP: `"rtsp_streaming_over_tcp": true` in `vst_config.json` (set during the VSS Warehouse step of either quick start).
6478845	Isaac/Omniverse. RTSP returns `has no caps` when two or more clients `DESCRIBE` the same stream during its cold bind window (right after Play). Reproducible with 3 parallel `ffprobe` — no VST needed.	Render-warm gate (already in the Halos repo) plus: do not run concurrent probes at startup. To relaunch the scenario on a live stack, use `closed-loop-testing/scripts/restart_isaac.sh`, which pauses VST stream processing across the boot window so its clients only `DESCRIBE` warm mounts.
6478903	VST/VIOS. Sensor `DELETE` returns 200 but leaks the live RTSP client (zombie). A zombie plus a re-added sensor double-`DESCRIBE` the same mount, which triggers 6478845 permanently. No config fix exists.	Do not delete and re-add sensors while streams are down. Restart `vss-vios-streamprocessing` to purge zombies — `closed-loop-testing/scripts/restart_isaac.sh` does this as part of a scenario relaunch. The Halos side now reuses matching sensors instead of delete-all on every launch.
6478175	Isaac/Omniverse (Replicator). Windowed UI with multi-camera RTSP leaks NVENC encode sessions across timeline stop/play. After ~3 replays it hits the 8-session GeForce cap, then floods `NVENCSTATUS=21` or crashes. Headless is unaffected.	Run headless. If you need the UI on a GeForce card, restart Isaac Sim between replays, or debug on a professional GPU.

Safety AI Monitor#

Accelerator backend availability#

On x86-64, Safety AI Monitor uses the GPU backend. On aarch64 IGX deployments, the launcher selects the PVA backend and requires the corresponding platform device access. If the selected accelerator backend is unavailable, run the Safety Core with Safety AI Monitor disabled: pass --mode skip to launch_hoisa.sh, or --saim-mode skip when starting the Safety Core container with a raw docker run.

Safe state recovery#

Latched safe state requires an explicit release#

Once the Safety Decision Maker latches a safe state, clearing the originating fault does not by itself resume normal operation. The latch is released only by a safe-release request from the command receiver endpoint. See 2.5 Clearing a Latched Safe State.
A restarted command receiver issues that request automatically on first contact with its peer, so a latch whose fault has already cleared is released without operator action. This is not a way out of a live fault: while any fault source is still active the SDM replies CMD_SAFE_RELEASE_DENIED and keeps the latch.
The latch lives in SDM process memory and is not persisted, so restarting the SDM discards it rather than releasing it through the handshake. A fault that is still present re-latches on the next evaluation; one that has cleared leaves no trace. Do not use an SDM restart as a recovery step, because it bypasses the handshake that tells the controller whether the release was granted.

AI Perception Provisioning#

Sensor provisioning issues#

Two open issues in the sensor provisioning chain leave AI Perception short of its cameras. They are upstream (VSS blueprint configurator and the sensor distribution service), not Halos Safety Core issues, and they apply to any profile that runs AI Perception. Neither recovers on its own; both are recoverable in place.

Bug ID	Symptom	Workaround
6511231	Blueprint configurator. Duplicate camera registration hot-loops, so perception reports duplicate camera names or a camera never registers at all. Restarting the sensor distribution service replays the stale event backlog and makes it worse.	Flush the event backlog, then register once — Recovering a camera that never registers.
6511265	Sensor distribution service. One camera sits at 0 FPS after re-registration and the active source count stays short, because perception holds the previous sensor uuid’s source bin while the live uuid is never added. For the 3D profile this corrupts the bird’s-eye output rather than degrading it.	Purge the stale source, then add the live one — Recovering a camera stuck at 0 FPS.

Recovering a camera that never registers#

Perception reports duplicate camera names, or one camera never appears no matter how often it is registered. Restarting the sensor distribution service makes it worse, because the service replays a Redis event stream that still holds every stale add and remove from earlier cycles, and those entries take the batch slots the live cameras need.

Flush the backlog, then register once. The order matters:

Warning

FLUSHALL empties every key in that Redis instance, not just the sensor event backlog. In the reference deployment Redis carries only perception state, but on a host where other services share the instance, take a backup or flush the sensor stream selectively instead.

docker exec redis redis-cli FLUSHALL
docker restart sdr-controller
docker restart vss-rtvi-cv

Wait about 15 seconds for the pipeline to come up empty, then run a single registration round:

docker exec isaac-sim bash -lc 'cd /isaac-sim && \
  ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --delete-all && sleep 3 && \
  ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --add-from-config /isaac-sim/sil/configs/cameras.yaml'

Perception reaches its full stream count in 30 to 60 seconds. If a camera is still missing, its stream is dead rather than unregistered: probe the Isaac Sim mount, and if it delivers no frames, relaunch the scenario with closed-loop-testing/scripts/restart_isaac.sh.

Recovering a camera stuck at 0 FPS#

One specific camera stays dead while the others run, perception logs No data from source ... trying reconnection for it, and the active source count sits one short. Perception is holding the sensor uuid from before the last re-registration, whose proxy mount no longer exists, and the live uuid was never added.

Identify which uuid is live and which one perception holds:

curl -s http://<HOST_IP>:30888/vst/api/v1/sensor/streams
curl -s http://localhost:9000/api/v1/stream/get-stream-info

The first lists each online sensor’s current proxy URL; the second shows what perception holds, where the camera reporting 0 FPS is the stale one. Remove it first — the batch is full, so an add before the remove fails:

curl -s -X POST http://localhost:9000/api/v1/stream/remove \
  -H 'Content-Type: application/json' \
  -d '{"key":"sensor","value":{"camera_id":"<stale-uuid>","camera_name":"<name>",
       "camera_url":"<the exact URL perception registered>","change":"camera_remove","metadata":{}}}'

Expect STREAM_REMOVE_SUCCESS. Two fields have to be exact: change must read camera_remove, or the call returns HTTP 400 and the stale source survives; and camera_url must be the URL perception registered it with, which is often an older proxy port rather than the current one. Recover that URL from the service log if the call returns No record found:

docker logs sdr-controller 2>&1 | grep <stale-uuid>

With the slot free, add the live uuid:

curl -s -X POST http://localhost:9000/api/v1/stream/add \
  -H 'Content-Type: application/json' \
  -d '{"key":"sensor","value":{"camera_id":"<live-uuid>","camera_name":"<name>",
       "camera_url":"<the live proxy URL>","change":"camera_add","metadata":{}}}'

Expect STREAM_ADD_SUCCESS, then the full active source count in the perception log.

Closed-Loop Testing Harness#

These constraints apply to the closed-loop testing harness — the sil and hil profiles — and not to a production deployment.

The simulated forklift does not act on the safety decision#

The forklift controller is a stimulus generator: it drives a fixed path and does not consume the safety command. Observe the decision through the in-scene safety indicator and the communication layer log, not through the forklift’s motion.

The communication layer endpoints are unauthenticated#

The communication layer binds its OPC UA server and its safety command port on all interfaces with no authentication and no source-address filter. Run the harness on a trusted network segment.

Both HIL hosts must be mutually routable#

The safety host pulls the camera streams from the stimulus host and sends the safety command back to it, so the two hosts need direct IP reachability in both directions. A safety host reachable only over a USB tether or through a jump host cannot be used without address translation.

HIL has no unified two-host orchestration#

The profile is configured through one environment file per host and brought up with one command per host, driven over ssh from a single seat. There is no single command that deploys both hosts.

The Safety AI Monitor is not part of the HIL profile#

The profile ships with the Safety Core launch mode set to skip, which runs the Safety Core without the Safety AI Monitor. Enabling it requires the live stream URLs and identifiers in the sensor configuration file, and adds load to a safety host that is already running perception.