> This page is for version 1.4.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 1.4.0 (Latest) (default): https://docs.nvidia.com/halos/outside-in-safety/1.4.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/halos/outside-in-safety/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/halos/outside-in-safety/_mcp/server. # Known Limitations > Current behavioural constraints of the Safety Core, the launchers and the closed-loop testing harness, each with its workaround. This page summarizes current behavioural constraints of the Safety Core and the tools around it. Each entry states the behaviour and what to do about it. ## ATL Use Case ### Single-forklift control model * The Automated Trailer Loading (ATL) reference application is designed and validated for controlling a single forklift operating within the monitored loading dock and trailer region. * If multiple forklifts enter the same Region of Interest (ROI) or trailer area concurrently, the Safety Event Integrator and Decision Maker may not be able to reliably associate safety control commands with the intended forklift under control, which can disrupt safe operation. ## Event Type Coverage ### Perception outputs the Safety Core uses * The MDX client turns these VSS outputs into Safety Core events: * from `mdx-events`: tripwire crossings and ROI entry and exit; * from `mdx-frames`: restricted-area and confined-area ROI violations and their clearing, social-distancing violations and object-pair distances, object presence, and co-movement. * The shipped mappings use tripwire crossings, restricted-area violations and object-pair distances. Other perception metadata is not used for decisions. * `alert_type: "empty_roi"` is accepted in a mapping rule, but the MDX client never produces it, so such a rule never matches. An empty ROI produces an `roi` clear instead. * See [Event Mapping Configuration](/halos/outside-in-safety/reference/event-mapping-configuration). ## Isaac Sim Streaming ### Upstream RTSP and VST issues Four issues were found on the Isaac Sim → VST stream path on Isaac Sim 6.0. Isaac Sim 6.1, which this release runs, ships the fix for the `has no caps` race below; the workarounds for the other three stay in place. They are **upstream** (Isaac/Omniverse and VST/VIOS), not Halos Safety Core issues, and they apply to any profile that runs Isaac Sim — 2D and 3D alike. Apply the workaround and continue. | Bug ID | Symptom | Workaround | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 6477478 | **Isaac/Omniverse.** RTSP over UDP: RTP delivery silently dies while the control plane stays healthy. Two modes — probe-triggered (\~4 min) and spontaneous (after hours). | Ingest over TCP: `"rtsp_streaming_over_tcp": true` in `vst_config.json` (set during the VSS Warehouse step of either quick start). | | 6478845 | **Isaac/Omniverse.** RTSP returns `has no caps` when two or more clients `DESCRIBE` the same stream during its cold bind window (right after Play). Reproducible with 3 parallel `ffprobe` — no VST needed. | Fixed upstream in Isaac Sim 6.1. On 6.0: render-warm gate (already in the Halos repo) plus: do not run concurrent probes at startup. To relaunch the scenario on a live stack, use `closed-loop-testing/scripts/restart_isaac.sh`, which pauses VST stream processing across the boot window so its clients only `DESCRIBE` warm mounts. | | 6478903 | **VST/VIOS.** Sensor `DELETE` returns 200 but leaks the live RTSP client (zombie). A zombie plus a re-added sensor double-`DESCRIBE` the same mount, which triggers 6478845 permanently. No config fix exists. | Do not delete and re-add sensors while streams are down. Restart `vss-vios-streamprocessing` to purge zombies — `closed-loop-testing/scripts/restart_isaac.sh` does this as part of a scenario relaunch. The Halos side now reuses matching sensors instead of delete-all on every launch. | | 6478175 | **Isaac/Omniverse (Replicator).** Windowed UI with multi-camera RTSP leaks NVENC encode sessions across timeline stop/play. After \~3 replays it hits the 8-session GeForce cap, then floods `NVENCSTATUS=21` or crashes. Headless is unaffected. | Run headless. If you need the UI on a GeForce card, restart Isaac Sim between replays, or debug on a professional GPU. | ## Safety AI Monitor ### Accelerator backend availability * On x86-64, the Safety AI Monitor (SAIM) uses the GPU. On aarch64 IGX deployments it uses the PVA and needs access to the platform's device nodes. If the accelerator is unavailable, run without SAIM: start the launcher with `--saim-mode skip`, and set `required_safety_monitor = 0` in `nvpss.conf` so that the SEI daemon does not require it. See [Monitor modes](/halos/outside-in-safety/reference/launcher#monitor-modes). ### GPU selection in the Docker runtime * **Behaviour.** With `launch_hoisa.sh --runtime docker`, or a plain `docker run` of the Safety Core image, `--gpu ` works only for `n = 0`. The container makes only GPU `n` visible to SAIM and also passes `--gpu `, so SAIM sees one GPU and any `n` of 1 or more stops it with `Error: --gpu is out of range (0-0)`. The container then stops. * **Workaround.** Leave out `--gpu`, so SAIM spreads its streams across the visible GPUs, or use `--gpu 0`. To pin SAIM to GPU 1 or higher, use `launch_hoisa.sh --runtime host --gpu `. ### A camera that is unreachable from the start is never reported failed * **Behaviour.** At startup SAIM reports every camera as `SENSOR_INVALID` with the start-up cause (cause 1), which an SDM does not count as a failure. If a camera's stream never connects, SAIM keeps retrying it, 5 s after each failed attempt (`[] RTSP connection failed, retrying every 5s`, logged on every 12th attempt), and keeps reporting the start-up cause for it. The camera is therefore never counted as failed, so the all-safety-relevant-sensors-failed latch cannot fire while that camera is in the reviewed sensor set. This holds even when no camera can be reached: SAIM stays registered and nothing latches through SAIM. When the camera comes up, SAIM starts watching it without a restart. * **Workaround.** Before you rely on active mode, confirm that every camera in `sensor_config.conf` streams, and check `/var/log/psf/saim.log` for `retrying every 5s`. ### SAIM analyses only 1920×1080 and 3840×2160 streams * **Behaviour.** SAIM divides each frame into a 15×15 grid of equal tiles, and its GPU and PVA kernels accept only frame sizes that fit that grid exactly. A camera of another size, such as 1280×720, 1920×1200, 2560×1440 or 4096×2160, is refused. A learning run logs a `[QA]` size error, such as `[QA] x is not compatible with the 15x15 analysis grid`, and learns no baseline for that camera. In active mode a baseline scores only the size it was learned at, so SAIM reports `SAIM_INTERNAL_ERROR` for such a camera and stops watching it. * **Workaround.** Stream the cameras SAIM watches at 1920×1080 or 3840×2160, and learn their baselines at that size. ### SAIM on NVStreamer URLs in a VSS deployment * **Behaviour.** NVStreamer serves a limited number of viewers per stream (`NUM_STREAMS` in the VSS `.env` file), and VST takes them when VSS starts. SAIM pointed at NVStreamer URLs in a VSS deployment can therefore be refused. * **Workaround.** Point `rtspUrl` in `sensor_config.conf` at the VST URLs, as the Deployment Guide does. ### SAIM and PCM do not reconnect after an SEI daemon restart * **Behaviour.** SAIM and the Perception Monitor (PCM) register with the SEI daemon only at startup. If the daemon restarts, they keep running but every heartbeat and report fails, and they never register again. The restarted daemon's presence report no longer lists them, so every SDM latches, and no release is accepted until they are back. The MDX client does not reconnect either: about 50 s after the daemon is lost it exits with `PSS heartbeat ACK failure limit reached — PSS presumed dead, exiting`. * **Workaround.** After the SEI daemon restarts, for any reason, restart the whole Safety Core with the launcher, then release the latched safe state from the command receiver. ## Perception Monitor ### PCM is required by default and can latch every SDM * **Behaviour.** The shipped `nvpss.conf` sets `required_perception_monitor = 1`. Every SDM latches on `AI_PIPELINE_INVALID` from PCM, and on PCM being lost. * **Workaround.** To run without PCM, start the launcher with `--pcm-mode skip` and set `required_perception_monitor = 0` in `nvpss.conf`. See [Monitor modes](/halos/outside-in-safety/reference/launcher#monitor-modes). ### A PCM baseline fits one decision maker configuration * **Behaviour.** PCM's baseline records the event bindings of the decision makers it was learned with. In an active run with other bindings, PCM does not use it and runs without its data check, logging `DVM: decision mapping changed for ; baseline not used, re-run LEARN`. The shipped reference baselines fit only `--app atl,pxc` with `atl_sdm_zone1.conf` and `pxc_sdm_forklift.conf` on the three reference cameras, so any other deployment, including the forklift safety function alone, needs a learn run first. * **Workaround.** Learn with the same `--app`, `--sdm-config` and `--event-mapping` as the active run, and check that `pcm.log` reports `data_validation=on`. See [Baselines](/halos/outside-in-safety/deployment/custom-deployment#saim-baselines). ### PCM turns a check off instead of reporting a setup problem * **Behaviour.** When Docker cannot be reached, the Kafka client reports an error, no usable PCM baseline remains, or no container in its list is found, PCM logs the cause and turns the affected check off. It does not report `AI_PIPELINE_INVALID` for this. Before its first verdict PCM sends nothing, so with both checks off from the start it never sends one. After its first verdict it keeps re-sending the last one, also when its checks turn off, so a standing `AI_PIPELINE_VALID` does not show that anything is still checked. A camera that never sends a frame, and a listed container that never appears, are only logged. * **Workaround.** After every start, check that both checks report results and that no missing-input or source-error warning appears, as in [Check that it is working](/halos/outside-in-safety/deployment/quick-deployment#check-that-it-is-working). See [Perception Monitor](/halos/outside-in-safety/integration/components/perception-monitor). ## Launchers ### Host accounts that share the container's group can reach the Safety Core sockets * **Behaviour.** When Safety Core processes on the host connect to the container, as with `launch_hoisa.sh --runtime docker`, `fsicom-agent` next to a container, or `--sdm-external`, the container mounts the host's `/run/nvpsf`, and the sockets admit any peer with the container user's group ID. A host account that can run with that group ID can connect to the sockets, or stand in for the NvPSD Gateway while its socket is not bound, and the SDMs and `fsicom-agent` on the host accept it. A host account with the container user's user ID owns the socket files and can delete them, so that processes connecting or reconnecting afterwards cannot reach the Safety Core, and the decision makers then hold their safe state. There is no dedicated Safety Core group that would close this. `launch_hoisa.sh --runtime docker` warns at every control-plane start when either ID is used on the host, but starts anyway. * **Workaround.** Do not give host accounts the container user's group ID, as primary group or through a host group, or its user ID. Check with the commands in [Host accounts with the container's group](/halos/outside-in-safety/integration#host-accounts-with-the-container-group). Or run every Safety Core process in the container without the `/run/nvpsf` mount, as in [Quick Deployment](/halos/outside-in-safety/deployment/quick-deployment). ### 3D files are not mounted into the container * **Behaviour.** `launch_hoisa.sh --runtime docker` mounts the camera list and `nvpss.conf` into the Safety Core container, but neither `sensor_pipelines_config.conf` nor the 3D sensor group file. In a 3D deployment the SEI daemon in the container cannot open the group file named by `sensorGroupConfig`, refuses to start, and the container stops. The MDX client in the container would also read the image's 2D `sensor_pipelines_config.conf` and drop every group event. The host's `--sensor-pipelines-config` reaches only the SDMs on the host. * **Workaround.** Start the container yourself with both files mounted at the paths the container reads; see [Container options](/halos/outside-in-safety/reference/launcher#container-options) and [3D perception](/halos/outside-in-safety/deployment/custom-deployment#cameras). Or run 3D deployments with `launch_hoisa.sh --runtime host`. ### `launch_hoisa.sh` has no combined forklift and proximity application * **Behaviour.** `launch_hoisa.sh` refuses `--app atl_proximity` with `Error: unsupported app 'atl_proximity'`. Only the container entrypoint starts `atl_sdm` together with `atl_proximity_sdm`. * **Workaround.** Use `--app atl,pxc` with the `atl_sdm_zone1.conf` and `pxc_sdm_forklift.conf` configurations, which runs `atl_sdm` and a `proximity_sdm` for the forklift–person pair: in the container, as in [Running more than one decision maker](/halos/outside-in-safety/deployment/custom-deployment#multiple-decision-makers), or with `launch_hoisa.sh --runtime host`. Or start the container directly with `--app atl_proximity`; see [Container options](/halos/outside-in-safety/reference/launcher#container-options). ### "unsupported command" warning on every safe-release response * **Behaviour.** The reference command receivers acknowledge every packet they receive, including the SDM's `CMD_SAFE_RELEASE_ACK` and `CMD_SAFE_RELEASE_DENIED`. The SDM does not accept those acknowledgements, so each release response adds `SDM: dropping PLC packet with unsupported command` to the SDM log. This is log noise; it does not affect the safe state. * **Workaround.** Ignore the warning. A PLC that does not acknowledge `0x09` and `0x0A` does not cause it. ## Safe state recovery ### Latched safe state requires an explicit release * Once the Safety Decision Maker latches a safe state, clearing the originating fault does not by itself resume normal operation. The latch is released only by a safe-release request from the command receiver endpoint. See [Release a latched safe state](/halos/outside-in-safety/deployment/quick-deployment#release-a-latched-safe-state). * A restarted command receiver issues that request automatically on first contact with its peer, so a latch whose fault has already cleared is released without operator action. This is not a way out of a live fault: while any fault source is still active the SDM replies `CMD_SAFE_RELEASE_DENIED` and keeps the latch. * The latch lives in SDM process memory and is not persisted, so restarting the SDM discards it rather than releasing it through the handshake. A fault that is still present re-latches on the next evaluation; one that has cleared leaves no trace. Do not use an SDM restart as a recovery step, because it bypasses the handshake that tells the controller whether the release was granted. ## AI Perception Provisioning ### Sensor provisioning issues Two open issues in the sensor provisioning chain leave AI Perception short of its cameras. They are **upstream** (VSS blueprint configurator and the sensor distribution service), not Halos Safety Core issues, and they apply to any profile that runs AI Perception. Neither recovers on its own; both are recoverable in place. | Bug ID | Symptom | Workaround | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | 6511231 | **Blueprint configurator.** Duplicate camera registration hot-loops, so perception reports duplicate camera names or a camera never registers at all. Restarting the sensor distribution service replays the stale event backlog and makes it worse. | Flush the event backlog, then register once — [Recovering a camera that never registers](#recovering-a-camera-that-never-registers). | | 6511265 | **Sensor distribution service.** One camera sits at 0 FPS after re-registration and the active source count stays short, because perception holds the previous sensor uuid's source bin while the live uuid is never added. For the 3D profile this corrupts the bird's-eye output rather than degrading it. | Purge the stale source, then add the live one — [Recovering a camera stuck at 0 FPS](#recovering-a-camera-stuck-at-0-fps). | ### Recovering a camera that never registers Perception reports duplicate camera names, or one camera never appears no matter how often it is registered. Restarting the sensor distribution service makes it worse, because the service replays a Redis event stream that still holds every stale add and remove from earlier cycles, and those entries take the batch slots the live cameras need. Flush the backlog, then register once. The order matters: > **Warning** > > `FLUSHALL` empties every key in that Redis instance, not just the sensor event > backlog. In the reference deployment Redis carries only perception state, but on a > host where other services share the instance, take a backup or flush the sensor > stream selectively instead. ```bash docker exec redis redis-cli FLUSHALL docker restart sdr-controller docker restart vss-rtvi-cv ``` Wait about 15 seconds for the pipeline to come up empty, then run a single registration round: ```bash docker exec isaac-sim bash -lc 'cd /isaac-sim && \ ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --delete-all && sleep 3 && \ ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --add-from-config /isaac-sim/sil/configs/cameras.yaml' ``` Perception reaches its full stream count in 30 to 60 seconds. If a camera is still missing, its stream is dead rather than unregistered: probe the Isaac Sim mount, and if it delivers no frames, relaunch the scenario with `closed-loop-testing/scripts/restart_isaac.sh`. ### Recovering a camera stuck at 0 FPS One specific camera stays dead while the others run, perception logs `No data from source ... trying reconnection` for it, and the active source count sits one short. Perception is holding the sensor uuid from before the last re-registration, whose proxy mount no longer exists, and the live uuid was never added. Identify which uuid is live and which one perception holds: ```bash curl -s http://:30888/vst/api/v1/sensor/streams curl -s http://localhost:9000/api/v1/stream/get-stream-info ``` The first lists each online sensor's current proxy URL; the second shows what perception holds, where the camera reporting 0 FPS is the stale one. Remove it first — the batch is full, so an add before the remove fails: ```bash curl -s -X POST http://localhost:9000/api/v1/stream/remove \ -H 'Content-Type: application/json' \ -d '{"key":"sensor","value":{"camera_id":"","camera_name":"", "camera_url":"","change":"camera_remove","metadata":{}}}' ``` Expect `STREAM_REMOVE_SUCCESS`. Two fields have to be exact: `change` must read `camera_remove`, or the call returns HTTP 400 and the stale source survives; and `camera_url` must be the URL perception registered it with, which is often an older proxy port rather than the current one. Recover that URL from the service log if the call returns `No record found`: ```bash docker logs sdr-controller 2>&1 | grep ``` With the slot free, add the live uuid: ```bash curl -s -X POST http://localhost:9000/api/v1/stream/add \ -H 'Content-Type: application/json' \ -d '{"key":"sensor","value":{"camera_id":"","camera_name":"", "camera_url":"","change":"camera_add","metadata":{}}}' ``` Expect `STREAM_ADD_SUCCESS`, then the full active source count in the perception log. ## Closed-Loop Testing Harness These constraints apply to the closed-loop testing harness — the `sil` and `hil` profiles — and not to a production deployment. ### The simulated forklift does not act on the ATL decision * The forklift controller is a stimulus generator: it drives a fixed path and does not consume the ATL safety command. Observe the decision through the in-scene safety indicator and the communication layer log, not through the forklift's motion. * Proximity decisions (`PSF_APP=pxc` or `both`) are the exception: the controller slows and stops the forklift on them. ### Proximity decisions are not addressed to a forklift * The Safety Core names no robot in a proximity decision, so every forklift on one decision topic obeys every decision. Give a second forklift its own `proximity.pair_topic` in `robots.yaml`, or disable proximity for it. * Proximity in the closed loop is SIL-only and runs on the 3D (Sparse4D) feed: on a 2D feed each camera's decision would reach the SDM separately, so one camera's NORMAL could override another's STOP. ### The Regression Testing Reporter scores ATL only * It subscribes to `/safety/command` and `/safety/is_muted`. On a `PSF_APP=pxc` run it reports `PSF_FEED_DEAD` and scores every frame as UNMUTED. ### The IGX Thor profiles run release 1.3 * `hil` and `base` on IGX Thor deploy VSS Warehouse 3.2.1 and the Safety Core 1.3 Thor package, and run ATL only. The 1.4 aarch64 packages are published, but the Thor launch script drives the 1.3 `launch_hoisa.sh` and has not been run against them, so the profiles stay on 1.3 for now. The Brev notebook also deploys VSS 3.2.1. ### The two-forklift scene is experimental * No calibration is published for `SCENARIO=warehouse_40x20`: with the shipped sample, perception runs the 20x20 geometry against a different building, so do not score its safety results. Build the scene's 2D dataset (both trailer bays) with `closed-loop-testing/isaac-sim/sil/calibration/build_vss_calibration.py` first, and run it on the 2D app: on this scene the 3D (Sparse4D) app stops detecting after seconds to minutes. ### The communication layer endpoints are unauthenticated * The communication layer binds its OPC UA server and its safety command port on all interfaces with no authentication and no source-address filter. Run the harness on a trusted network segment. ### Both HIL hosts must be mutually routable * The safety host pulls the camera streams from the stimulus host and sends the safety command back to it, so the two hosts need direct IP reachability in both directions. A safety host reachable only over a USB tether or through a jump host cannot be used without address translation. ### HIL has no unified two-host orchestration * The profile is configured through one environment file per host and brought up with one command per host, driven over ssh from a single seat. There is no single command that deploys both hosts. ### The Safety AI Monitor is not part of the HIL profile * The profile ships with the Safety Core launch mode set to `skip`, which runs the Safety Core without the Safety AI Monitor. Enabling it requires the live stream URLs and identifiers in the sensor configuration file, and adds load to a safety host that is already running perception.