Safe-State Model
Every reference Safety Decision Maker (SDM) follows one safe-state model. This page explains it: what puts an SDM into the safe state, what each cause needs before it clears, and how the PLC or command receiver releases the latch. How to implement the model in your own SDM is on Writing a decision maker. The numeric cause values and the audit records are on Causes and Records.
Causes, latch and output
The model has three parts:
- Causes. Each fault condition the SDM watches is a cause, and each cause is either active or not.
- The latch. Raising any latching cause sets the latch. While the latch is set, the SDM sends its safe output instead of its decision, and repeats it.
- The release. The latch clears only when the SDM accepts a release request from the PLC, and it accepts one only while no cause is active.
Two rules follow, and they are the core of the model:
- A cause clears only on its own evidence. A lost monitor clears when that monitor is back, not when other traffic arrives. Ordinary traffic never stands in for the evidence a cause is waiting for.
- A cause clearing never clears the latch. The SDM logs
... latch awaits PLC releaseand keeps the safe output until an operator, through the PLC, confirms that the area is safe.
The latch is held in the SDM process. A restarted SDM starts with no latch and no active cause, then relearns the standing conditions from the SEI (see Re-assertion of standing verdicts). Its freshness and gateway-heartbeat checks start at launch, so an SDM that receives nothing still latches after about 7 seconds.
Cause categories
The first five rows are sub-causes of one SEI cause. Each SW_FAIL in a request raises its own
sub-cause, whatever its position in the request and whatever the mode, so a request carrying two
faults leaves both active until each has its own evidence.
What puts the SEI into ERROR: delivery to the gateway failing past its retry budget, a
must-deliver report refused because its queue is full, a processing stage failing without
interruption for fusion_failure_error_ms, or SAIM’s heartbeat budget running out. SAIM’s
heartbeat above the warn tier puts it into DEGRADED, which the SDMs treat like NORMAL.
Conditions that do not latch
Some conditions affect decisions only while they last:
- One unhealthy sensor. Entries from a pipeline with a standing
SENSOR_INVALIDarrive withisHealthy = false, and the SDM skips them. The SDM keeps deciding on the other sensors, and latches only when every sensor in the reviewed set has failed. - Untrusted entries. Entries with
isTrustedSource = falseare skipped. TheAI_PIPELINE_INVALIDthat causes this also latches, as above. STALEentries. Entries older than the SEI’stimeWindowSize, or dated more than that far in the future, are dropped.- Gateway heartbeat, tiers 1 and 2. Tier 1 only warns; tier 2 holds the safe output without latching.
- FSI cold boot. Until the FSI SDM has received its first decision and its first CCPLEX heartbeat after boot, it holds the safe output without latching, so it does not need a release before the CCPLEX side is up. A loss after that latches.
Sensor health
Each CCPLEX SDM reads its reviewed sensor set from sensor_pipelines_config.conf. In 2D the set is
the cameras; in 3D it is the BEV group alone. The FSI SDMs use a compiled 2D set.
SAIM reports every camera as SENSOR_INVALID with CAUSE_SAIM_INIT when it starts, before the
first frame. That seed does not latch, and it marks the pipeline as initialising until the pipeline
next reports SENSOR_VALID, or healthy and trusted. While SAIM initialises, the SEI can mark the
camera’s other events unhealthy with their own cause; those entries with isHealthy = false are
not counted as failed for an initialising pipeline. A SENSOR_INVALID with any other cause,
including one the SDM does not recognise, counts as a real failure, also while the pipeline is
initialising. Every SDM, on the CCPLEX and on the FSI, applies this rule, and clears the
initialising state when it starts.
A pipeline id outside the reviewed set is logged once and does not affect the latch. A set that lists a camera that never reports can never be all-failed, so the latch can never fire; a set that lists fewer cameras than are deployed fires while a working camera is still delivering. Match the file to the cameras the deployment runs. Each SDM logs the set it loaded at start-up.
In 3D the SEI derives the group’s verdict from its member cameras: the group is valid while any member is valid, invalid only when every member is invalid and none is still initialising.
Reading active_causes
A denied release logs the active causes as a decimal bitmask, for example
ATL: PLC safe-release denied; active safe-state fault source remains, active_causes=10. The bit
values differ between atl_sdm and the other SDMs; both sets are in
SDM latch causes. For atl_sdm, 10 is 8 + 2:
decision freshness and an SEI sub-cause. The SDM’s log names the sub-cause when it is raised and
when it clears (ATL: PSS sub-cause cleared: <name>; latch awaits PLC release).
Re-assertion of standing verdicts
Monitors report a verdict when it changes. An SDM that starts, restarts or reconnects after that change would never hear it. Three mechanisms close the gap:
The SEI keeps its verdicts across a monitor’s loss; only the monitor’s next report changes them. A
restarted PCM always sends its first verdict, so its first AI_PIPELINE_VALID clears an
AI_PIPELINE_INVALID that the previous PCM left; the latch still needs a release. The verdicts are
lost when the SEI itself restarts. SAIM repeats its verdicts every 30 seconds and PCM every
5 seconds (verdict_resend_ms), but both register only at start and do not register with a
restarted SEI daemon, so restart them with it.
The release protocol from the controller side
A PLC or command receiver that drives the release must follow this exchange. The packet format and the SDM’s checks are on Writing a decision maker.
-
The SDM latches and sends its safe output, ending in
CMD_SW_ERROR(0x03). It repeats it on every decision request and on its periodic repeat. -
The controller holds the machine in its safe state and shows the operator that a release is possible. The reference receivers offer it once they have received
CMD_SW_ERRORorCMD_SAFE_RELEASE_DENIED. -
The operator confirms that the area is safe. The controller sends
CMD_SAFE_RELEASE_REQUEST(0x08) from the address and port the SDM is configured with (--cmd_rx_ip,--cmd_rx_port), with the application’s identifier and a valid CRC. -
The SDM answers:
-
On
CMD_SAFE_RELEASE_ACK, the controller resumes acting on decisions. OnCMD_SAFE_RELEASE_DENIED, it stays in the safe state; the SDM log names the active causes.
An accepted release resets the SDM’s view of the scene. atl_sdm then treats the trailer as empty:
no forklift, no person, no restricted-area violation. Release only when the forklift is outside
the trailer and the restricted area, and no person is inside either. The reference receivers send
one release on their own about 500 ms after they first hear from the SDM, so restarting a receiver
resets the scene in the same way, and needs the same check.
The SDM checks only its own causes. It cannot see the work area, so a release is the operator’s
statement that the area is safe. A CMD_SAFE_RELEASE_DENIED usually means the original fault is
still present, not that the scene is unsafe.
The reference command receivers
What the operator sees
On the command receiver console:
The per-packet detail goes to the receiver’s log file, /tmp/cmdrx.log by default.
In the SDM’s log, with the prefix ATL:, Proximity: or APX::
The full list of SEI and gateway records is on Causes and Records. Where each log file is depends on how the Safety Core was started; see Logs.