Known Limitations

This page summarizes current behavioural constraints of the Safety Core and the tools around it. Each entry states the behaviour and what to do about it.

ATL Use Case

Single-forklift control model

• The Automated Trailer Loading (ATL) reference application is designed and validated for controlling a single forklift operating within the monitored loading dock and trailer region.
• If multiple forklifts enter the same Region of Interest (ROI) or trailer area concurrently, the Safety Event Integrator and Decision Maker may not be able to reliably associate safety control commands with the intended forklift under control, which can disrupt safe operation.

Event Type Coverage

Perception outputs the Safety Core uses

• The MDX client turns these VSS outputs into Safety Core events:
  • from mdx-events: tripwire crossings and ROI entry and exit;
  • from mdx-frames: restricted-area and confined-area ROI violations and their clearing, social-distancing violations and object-pair distances, object presence, and co-movement.
• The shipped mappings use tripwire crossings, restricted-area violations and object-pair distances. Other perception metadata is not used for decisions.
• alert_type: "empty_roi" is accepted in a mapping rule, but the MDX client never produces it, so such a rule never matches. An empty ROI produces an roi clear instead.
• See Event Mapping Configuration.

Isaac Sim Streaming

Upstream RTSP and VST issues

Four issues were found on the Isaac Sim → VST stream path on Isaac Sim 6.0. Isaac Sim 6.1, which this release runs, ships the fix for the has no caps race below; the workarounds for the other three stay in place. They are upstream (Isaac/Omniverse and VST/VIOS), not Halos Safety Core issues, and they apply to any profile that runs Isaac Sim — 2D and 3D alike. Apply the workaround and continue.

Safety AI Monitor

Accelerator backend availability

• On x86-64, the Safety AI Monitor (SAIM) uses the GPU. On aarch64 IGX deployments it uses the PVA and needs access to the platform’s device nodes. If the accelerator is unavailable, run without SAIM: start the launcher with --saim-mode skip, and set required_safety_monitor = 0 in nvpss.conf so that the SEI daemon does not require it. See Monitor modes.

GPU selection in the Docker runtime

• Behaviour. With launch_hoisa.sh --runtime docker, or a plain docker run of the Safety Core image, --gpu <n> works only for n = 0. The container makes only GPU n visible to SAIM and also passes --gpu <n>, so SAIM sees one GPU and any n of 1 or more stops it with Error: --gpu <n> is out of range (0-0). The container then stops.
• Workaround. Leave out --gpu, so SAIM spreads its streams across the visible GPUs, or use --gpu 0. To pin SAIM to GPU 1 or higher, use launch_hoisa.sh --runtime host --gpu <n>.

A camera that is unreachable from the start is never reported failed

• Behaviour. At startup SAIM reports every camera as SENSOR_INVALID with the start-up cause (cause 1), which an SDM does not count as a failure. If a camera’s stream never connects, SAIM keeps retrying it, 5 s after each failed attempt ([<camera>] RTSP connection failed, retrying every 5s, logged on every 12th attempt), and keeps reporting the start-up cause for it. The camera is therefore never counted as failed, so the all-safety-relevant-sensors-failed latch cannot fire while that camera is in the reviewed sensor set. This holds even when no camera can be reached: SAIM stays registered and nothing latches through SAIM. When the camera comes up, SAIM starts watching it without a restart.
• Workaround. Before you rely on active mode, confirm that every camera in sensor_config.conf streams, and check /var/log/psf/saim.log for retrying every 5s.

SAIM analyses only 1920×1080 and 3840×2160 streams

• Behaviour. SAIM divides each frame into a 15×15 grid of equal tiles, and its GPU and PVA kernels accept only frame sizes that fit that grid exactly. A camera of another size, such as 1280×720, 1920×1200, 2560×1440 or 4096×2160, is refused. A learning run logs a [QA] size error, such as [QA] <w>x<h> is not compatible with the 15x15 analysis grid, and learns no baseline for that camera. In active mode a baseline scores only the size it was learned at, so SAIM reports SAIM_INTERNAL_ERROR for such a camera and stops watching it.
• Workaround. Stream the cameras SAIM watches at 1920×1080 or 3840×2160, and learn their baselines at that size.

SAIM on NVStreamer URLs in a VSS deployment

• Behaviour. NVStreamer serves a limited number of viewers per stream (NUM_STREAMS in the VSS .env file), and VST takes them when VSS starts. SAIM pointed at NVStreamer URLs in a VSS deployment can therefore be refused.
• Workaround. Point rtspUrl in sensor_config.conf at the VST URLs, as the Deployment Guide does.

SAIM and PCM do not reconnect after an SEI daemon restart

• Behaviour. SAIM and the Perception Monitor (PCM) register with the SEI daemon only at startup. If the daemon restarts, they keep running but every heartbeat and report fails, and they never register again. The restarted daemon’s presence report no longer lists them, so every SDM latches, and no release is accepted until they are back. The MDX client does not reconnect either: about 50 s after the daemon is lost it exits with PSS heartbeat ACK failure limit reached — PSS presumed dead, exiting.
• Workaround. After the SEI daemon restarts, for any reason, restart the whole Safety Core with the launcher, then release the latched safe state from the command receiver.

Perception Monitor

PCM is required by default and can latch every SDM

• Behaviour. The shipped nvpss.conf sets required_perception_monitor = 1. Every SDM latches on AI_PIPELINE_INVALID from PCM, and on PCM being lost.
• Workaround. To run without PCM, start the launcher with --pcm-mode skip and set required_perception_monitor = 0 in nvpss.conf. See Monitor modes.

A PCM baseline fits one decision maker configuration

• Behaviour. PCM’s baseline records the event bindings of the decision makers it was learned with. In an active run with other bindings, PCM does not use it and runs without its data check, logging DVM: decision mapping changed for <camera>; baseline not used, re-run LEARN. The shipped reference baselines fit only --app atl,pxc with atl_sdm_zone1.conf and pxc_sdm_forklift.conf on the three reference cameras, so any other deployment, including the forklift safety function alone, needs a learn run first.
• Workaround. Learn with the same --app, --sdm-config and --event-mapping as the active run, and check that pcm.log reports data_validation=on. See Baselines.

PCM turns a check off instead of reporting a setup problem

• Behaviour. When Docker cannot be reached, the Kafka client reports an error, no usable PCM baseline remains, or no container in its list is found, PCM logs the cause and turns the affected check off. It does not report AI_PIPELINE_INVALID for this. Before its first verdict PCM sends nothing, so with both checks off from the start it never sends one. After its first verdict it keeps re-sending the last one, also when its checks turn off, so a standing AI_PIPELINE_VALID does not show that anything is still checked. A camera that never sends a frame, and a listed container that never appears, are only logged.
• Workaround. After every start, check that both checks report results and that no missing-input or source-error warning appears, as in Check that it is working. See Perception Monitor.

Launchers

Host accounts that share the container’s group can reach the Safety Core sockets

• Behaviour. When Safety Core processes on the host connect to the container, as with launch_hoisa.sh --runtime docker, fsicom-agent next to a container, or --sdm-external, the container mounts the host’s /run/nvpsf, and the sockets admit any peer with the container user’s group ID. A host account that can run with that group ID can connect to the sockets, or stand in for the NvPSD Gateway while its socket is not bound, and the SDMs and fsicom-agent on the host accept it. A host account with the container user’s user ID owns the socket files and can delete them, so that processes connecting or reconnecting afterwards cannot reach the Safety Core, and the decision makers then hold their safe state. There is no dedicated Safety Core group that would close this. launch_hoisa.sh --runtime docker warns at every control-plane start when either ID is used on the host, but starts anyway.
• Workaround. Do not give host accounts the container user’s group ID, as primary group or through a host group, or its user ID. Check with the commands in Host accounts with the container’s group. Or run every Safety Core process in the container without the /run/nvpsf mount, as in Quick Deployment.

3D files are not mounted into the container

• Behaviour. launch_hoisa.sh --runtime docker mounts the camera list and nvpss.conf into the Safety Core container, but neither sensor_pipelines_config.conf nor the 3D sensor group file. In a 3D deployment the SEI daemon in the container cannot open the group file named by sensorGroupConfig, refuses to start, and the container stops. The MDX client in the container would also read the image’s 2D sensor_pipelines_config.conf and drop every group event. The host’s --sensor-pipelines-config reaches only the SDMs on the host.
• Workaround. Start the container yourself with both files mounted at the paths the container reads; see Container options and 3D perception. Or run 3D deployments with launch_hoisa.sh --runtime host.

launch_hoisa.sh has no combined forklift and proximity application

• Behaviour. launch_hoisa.sh refuses --app atl_proximity with Error: unsupported app 'atl_proximity'. Only the container entrypoint starts atl_sdm together with atl_proximity_sdm.
• Workaround. Use --app atl,pxc with the atl_sdm_zone1.conf and pxc_sdm_forklift.conf configurations, which runs atl_sdm and a proximity_sdm for the forklift–person pair: in the container, as in Running more than one decision maker, or with launch_hoisa.sh --runtime host. Or start the container directly with --app atl_proximity; see Container options.

”unsupported command” warning on every safe-release response

• Behaviour. The reference command receivers acknowledge every packet they receive, including the SDM’s CMD_SAFE_RELEASE_ACK and CMD_SAFE_RELEASE_DENIED. The SDM does not accept those acknowledgements, so each release response adds SDM: dropping PLC packet with unsupported command to the SDM log. This is log noise; it does not affect the safe state.
• Workaround. Ignore the warning. A PLC that does not acknowledge 0x09 and 0x0A does not cause it.

Safe state recovery

Latched safe state requires an explicit release

• Once the Safety Decision Maker latches a safe state, clearing the originating fault does not by itself resume normal operation. The latch is released only by a safe-release request from the command receiver endpoint. See Release a latched safe state.
• A restarted command receiver issues that request automatically on first contact with its peer, so a latch whose fault has already cleared is released without operator action. This is not a way out of a live fault: while any fault source is still active the SDM replies CMD_SAFE_RELEASE_DENIED and keeps the latch.
• The latch lives in SDM process memory and is not persisted, so restarting the SDM discards it rather than releasing it through the handshake. A fault that is still present re-latches on the next evaluation; one that has cleared leaves no trace. Do not use an SDM restart as a recovery step, because it bypasses the handshake that tells the controller whether the release was granted.

AI Perception Provisioning

Sensor provisioning issues

Two open issues in the sensor provisioning chain leave AI Perception short of its cameras. They are upstream (VSS blueprint configurator and the sensor distribution service), not Halos Safety Core issues, and they apply to any profile that runs AI Perception. Neither recovers on its own; both are recoverable in place.

Recovering a camera that never registers

Perception reports duplicate camera names, or one camera never appears no matter how often it is registered. Restarting the sensor distribution service makes it worse, because the service replays a Redis event stream that still holds every stale add and remove from earlier cycles, and those entries take the batch slots the live cameras need.

Flush the backlog, then register once. The order matters:

docker exec redis redis-cli FLUSHALL
docker restart sdr-controller
docker restart vss-rtvi-cv

Wait about 15 seconds for the pipeline to come up empty, then run a single registration round:

docker exec isaac-sim bash -lc 'cd /isaac-sim && \
  ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --delete-all && sleep 3 && \
  ./python.sh /isaac-sim/sil/scripts/vst_sensor_manager.py --add-from-config /isaac-sim/sil/configs/cameras.yaml'

Perception reaches its full stream count in 30 to 60 seconds. If a camera is still missing, its stream is dead rather than unregistered: probe the Isaac Sim mount, and if it delivers no frames, relaunch the scenario with closed-loop-testing/scripts/restart_isaac.sh.

Recovering a camera stuck at 0 FPS

One specific camera stays dead while the others run, perception logs No data from source ... trying reconnection for it, and the active source count sits one short. Perception is holding the sensor uuid from before the last re-registration, whose proxy mount no longer exists, and the live uuid was never added.

Identify which uuid is live and which one perception holds:

curl -s http://<HOST_IP>:30888/vst/api/v1/sensor/streams
curl -s http://localhost:9000/api/v1/stream/get-stream-info

The first lists each online sensor’s current proxy URL; the second shows what perception holds, where the camera reporting 0 FPS is the stale one. Remove it first — the batch is full, so an add before the remove fails:

curl -s -X POST http://localhost:9000/api/v1/stream/remove \
  -H 'Content-Type: application/json' \
  -d '{"key":"sensor","value":{"camera_id":"<stale-uuid>","camera_name":"<name>",
       "camera_url":"<the exact URL perception registered>","change":"camera_remove","metadata":{}}}'

Expect STREAM_REMOVE_SUCCESS. Two fields have to be exact: change must read camera_remove, or the call returns HTTP 400 and the stale source survives; and camera_url must be the URL perception registered it with, which is often an older proxy port rather than the current one. Recover that URL from the service log if the call returns No record found:

docker logs sdr-controller 2>&1 | grep <stale-uuid>

With the slot free, add the live uuid:

curl -s -X POST http://localhost:9000/api/v1/stream/add \
  -H 'Content-Type: application/json' \
  -d '{"key":"sensor","value":{"camera_id":"<live-uuid>","camera_name":"<name>",
       "camera_url":"<the live proxy URL>","change":"camera_add","metadata":{}}}'

Expect STREAM_ADD_SUCCESS, then the full active source count in the perception log.

Closed-Loop Testing Harness

These constraints apply to the closed-loop testing harness — the sil and hil profiles — and not to a production deployment.

The simulated forklift does not act on the ATL decision

• The forklift controller is a stimulus generator: it drives a fixed path and does not consume the ATL safety command. Observe the decision through the in-scene safety indicator and the communication layer log, not through the forklift’s motion.
• Proximity decisions (PSF_APP=pxc or both) are the exception: the controller slows and stops the forklift on them.

Proximity decisions are not addressed to a forklift

• The Safety Core names no robot in a proximity decision, so every forklift on one decision topic obeys every decision. Give a second forklift its own proximity.pair_topic in robots.yaml, or disable proximity for it.
• Proximity in the closed loop is SIL-only and runs on the 3D (Sparse4D) feed: on a 2D feed each camera’s decision would reach the SDM separately, so one camera’s NORMAL could override another’s STOP.

The Regression Testing Reporter scores ATL only

• It subscribes to /safety/command and /safety/is_muted. On a PSF_APP=pxc run it reports PSF_FEED_DEAD and scores every frame as UNMUTED.

The IGX Thor profiles run release 1.3

• hil and base on IGX Thor deploy VSS Warehouse 3.2.1 and the Safety Core 1.3 Thor package, and run ATL only. The 1.4 aarch64 packages are published, but the Thor launch script drives the 1.3 launch_hoisa.sh and has not been run against them, so the profiles stay on 1.3 for now. The Brev notebook also deploys VSS 3.2.1.

The two-forklift scene is experimental

• No calibration is published for SCENARIO=warehouse_40x20: with the shipped sample, perception runs the 20x20 geometry against a different building, so do not score its safety results. Build the scene’s 2D dataset (both trailer bays) with closed-loop-testing/isaac-sim/sil/calibration/build_vss_calibration.py first, and run it on the 2D app: on this scene the 3D (Sparse4D) app stops detecting after seconds to minutes.

The communication layer endpoints are unauthenticated

• The communication layer binds its OPC UA server and its safety command port on all interfaces with no authentication and no source-address filter. Run the harness on a trusted network segment.

Both HIL hosts must be mutually routable

• The safety host pulls the camera streams from the stimulus host and sends the safety command back to it, so the two hosts need direct IP reachability in both directions. A safety host reachable only over a USB tether or through a jump host cannot be used without address translation.

HIL has no unified two-host orchestration

• The profile is configured through one environment file per host and brought up with one command per host, driven over ssh from a single seat. There is no single command that deploys both hosts.

The Safety AI Monitor is not part of the HIL profile

• The profile ships with the Safety Core launch mode set to skip, which runs the Safety Core without the Safety AI Monitor. Enabling it requires the live stream URLs and identifiers in the sensor configuration file, and adds load to a safety host that is already running perception.