Skip to navigation
Safety Core Components

Safety AI Monitor (SAIM)

View as Markdown

The Safety AI Monitor (SAIM, safety_monitor) judges whether each camera’s video can be trusted. It reads the same RTSP streams as the perception pipeline, compares every frame with a baseline learned for that camera, and reports a per-camera verdict, SENSOR_INVALID or SENSOR_VALID, to the SEI daemon. SEI uses the verdicts to mark events from an untrusted camera, and the SDMs use them for their all-sensors-failed check.

SAIM process: for each camera an RTSP client, NVDEC decoder and frame analyzer feed a frame scorer that uses the camera's baseline and thresholds; a debounce counter and the verdict reporter send SENSOR_INVALID or SENSOR_VALID through the SEI client

SAIM is required by default. The shipped nvpss.conf sets required_safety_monitor = 1. When a required SAIM never registers, or is lost after registering, SEI raises a SW_FAIL with cause 15 and every SDM latches its safe state. Set the key to 0 only for a deployment without SAIM, and then start the launchers with --saim-mode skip; see SEI configuration.

Under the Docker runtime SAIM runs inside the Safety Core container. Under --runtime host it runs on the host.

Modes

ModeWhat SAIM does
LEARNReads every configured camera for --learn-duration seconds and writes one baseline file per camera. It does not register with SEI and reports no verdict. Run it at installation, and again when a camera is moved or its scene changes.
ACTIVELoads the baselines, registers with SEI, checks every frame and reports verdicts.

Both launchers select the mode with --saim-mode learn|active|skip; skip does not start SAIM. launch_hoisa.sh --mode is a deprecated alias for --saim-mode. The Perception Monitor has its own --pcm-mode. When the launchers start the control plane, required_safety_monitor in nvpss.conf must be 1 for --saim-mode active and 0 for learn or skip. A learning SAIM stops when its baselines are written and never switches to ACTIVE. See Monitor modes.

Command line

OptionDefaultMeaning
--mode <mode>ACTIVELEARN or ACTIVE, upper case only; learn is rejected.
--sensor-config <file>— (required)The camera list. See Inputs.
--threshold-config <file>thresholds.cfg in the working directoryScoring thresholds, ACTIVE only. See Thresholds.
--learn-duration <sec>300LEARN observation time, 1–3600.
--gpu <id>round-robinRun every camera on this GPU. See GPU selection.

SAIM has no --help and no baseline-directory option: it reads and writes baselines in its working directory, and the launchers change into the baseline directory before they start it. An unknown argument prints Error: unknown argument: <arg> and exits 1. SAIM exits 0 only when every camera’s pipeline ended without error.

Inputs

SAIM reads the cameras from sensor_config.conf: one line per camera with pipelineId, sensorName and rtspUrl. The format and limits are in Sensor files.

ItemBehaviour
RTSP controlTCP over IPv4, 10 s timeouts. The URL must start with rtsp://; the default port is 554. No authentication is supported. SAIM sends DESCRIBE to the rtspUrl, then sends PLAY, keep-alives and TEARDOWN to the control URL in the server’s answer (the session-level a=control, resolved against Content-Base), so a server that answers with a per-viewer URL, such as NVStreamer, works.
RTSP mediaRTP over UDP unicast. A firewall between SAIM and the camera or VST must allow it.
CodecH.264 only, decoded on the GPU with NVDEC. Frames before the first IDR frame are dropped.
Frame size1920×1080 or 3840×2160. The analysis divides each frame into a 15×15 grid of equal tiles, and the GPU and PVA kernels accept only sizes that fit it exactly. Other common sizes, such as 1280×720, 1920×1200, 2560×1440 and 4096×2160, are refused. In LEARN SAIM logs a [QA] size error, such as [QA] <w>x<h> is not compatible with the 15x15 analysis grid ([QA-PVA] on the PVA), and learns no baseline for that camera. In ACTIVE the stream’s size differs from the baseline’s, so SAIM stops that camera with SAIM_INTERNAL_ERROR; see Camera failures.
Session endThe session ends after more than 5 s without packets, which SAIM notices within about 8 s, or on an RTCP BYE.

What SAIM checks

SAIM computes 12 image metrics on every frame: histogram mean and variance, RMS contrast, edge density, entropy, Laplacian variance (sharpness), the mean and variance of the two colour channels, and saturation mean and variance. It also measures each tile of a 15×15 grid over the frame. The GPU or PVA only measures; the scoring runs on the CPU and gives three scores from 0 to 100:

  • Brightness and structure, from the brightness, contrast, edge, entropy and sharpness metrics, and colour, from the colour metrics. Each metric is scored against the camera’s baseline, but a change counts only as far as the tiles agree on it beyond chance. A forklift or a crowd that changes a few tiles therefore does not lower these scores; a change across the whole lens does.
  • Area, from still tiles that have newly turned black or white, lost fine detail, or gained colour noise compared with the baseline, for about 5 s. Tiles with movement are skipped. With the default thresholds the area score reaches the low threshold, 45, when about 2.5% of the frame is affected, and 0 at about 4.5%.

The frame score is the lowest of the three.

Because the baseline captures how that camera normally looks, a sustained low score points to problems such as:

  • out-of-distribution frames whose image statistics leave the learned range;
  • a blocked or partly covered lens, including a covered area of a few percent of the frame;
  • fogging or condensation;
  • loss of focus;
  • abnormal brightness, contrast or colour.

SAIM also reports a lost stream, dropped video packets and its own analysis failures.

To check these responses on a test deployment, inject camera faults with SAIM distortion injection.

The analysis runs on the GPU on x86-64 and on the PVA on aarch64. The backend is fixed when SAIM is built. Both backends need CUDA and NVDEC, and both give the same verdicts.

Baselines

ItemBehaviour
File name<sensorName>_baseline.cfg in the working directory. Renaming a camera therefore needs a new baseline.
ContentsThe camera’s whole-frame metrics, its frame size and a profile of every tile. A baseline scores only the frame size it was learned at: in ACTIVE a stream of another size stops that camera with [<sensor>] profile was learned at <w>x<h> but the stream is <w>x<h>; re-run LEARN for this camera.
Earlier releasesBaselines learned by an earlier release have no tile profile and are refused: [Config] BaselineValues: <file> was learned on a 0x0 tile grid but this build uses 15x15. Re-run LEARN to regenerate this baseline. Learn every camera again after upgrading. A learned .cfg is used before the shipped .cfg.default, so an old learned file is not replaced by the new template; delete it or learn again.
Default directory/opt/nvidia/psf/bin/saim_baseline, set with launch_hoisa.sh --baseline-dir. Under the Docker runtime the launcher mounts this host directory into the container, writable in LEARN and read-only in ACTIVE, so baselines persist from LEARN to ACTIVE.
Shipped defaultsCamera_baseline.cfg.default, Camera_01_baseline.cfg.default and Camera_02_baseline.cfg.default, learned on the reference warehouse scene. They match only those camera names.
Lookup in ACTIVEA learned .cfg wins. Without it SAIM uses .cfg.default and logs Notice: baseline '<file>' not found for sensor '<name>'; falling back to default. With neither, SAIM exits 1 before registering.
LEARN resultA baseline is saved only when the camera delivered the planned number of frames, at least 30, and the outlier filter kept enough of them. Otherwise the log says why and nothing is saved. In LEARN, six failed connection attempts, or more than five interruptions of a stream, stop every camera; a gap of more than 5 s without packets counts as an interruption. [<sensor>] LEARN: tiles: <n> series too short for a band; run LEARN longer means the run was too short for some tile checks: the baseline is saved, but those checks can never alert for that camera. Learn again for longer.

The shipped defaults describe the reference scene only. A site with other cameras or another scene must run LEARN; see Custom deployment.

Thresholds

ACTIVE reads thresholds.cfg. The launchers pass /opt/nvidia/psf/bin/thresholds.cfg, and the container launcher refuses to start SAIM when that file is missing. launch_hoisa.sh --threshold-config <file> replaces it.

KeyDefaultMeaning
w_histogram, w_contrast, w_edge, w_entropy, w_laplacian0.25, 0.20, 0.20, 0.20, 0.15Weights of the brightness and structure metrics.
uv_weight_cb, uv_weight_cr, uv_weight_sat0.4, 0.4, 0.2Weights of the colour metrics.
k, sigma_floor_fraction, score_at_k_sigma4.0, 0.0001, 55How far from the baseline a metric may move before it scores low.
canny_low_thresh, canny_high_thresh50, 150Edge detector thresholds.
score_low_threshold, score_high_threshold45, 55Frame scores below 45 count as bad, above 55 as good. score_low_threshold also sets where the area score reaches 0 (2.5% of the frame divided by 1 − threshold/100, about 4.5% at the default) and, with k and score_at_k_sigma, how far a tile may move from its baseline. The area score crosses the low threshold at 2.5% of the frame whatever its value.
counter_max, max_increment, max_decrement225, 45, 9Debounce counter; see Verdicts and timing.

The shipped file holds the built-in defaults. When the file is missing or fails validation, SAIM logs WARNING: threshold config <path> is missing or invalid; using built-in defaults for each camera and continues.

Verdicts and timing

SAIM reports two verdicts per camera, each with a cause. The cause values are listed in Causes and records.

CauseVerdictReported when
SAIM_INITSENSOR_INVALIDOnce per camera at registration, before any frame is analysed. Every camera starts untrusted.
SAIM_INPUT_DEGRADEDSENSOR_INVALIDThe frame score stays low (see below), or the camera sent a keyframe but did not reach SENSOR_VALID within 15 s.
SAIM_STREAM_DISCONNECTSENSOR_INVALIDNo keyframe within 30 s of the first packet; or, for a trusted camera, its stream ends (more than 5 s without packets, or an RTCP BYE) or a connection attempt fails. It is sent before SAIM reconnects.
SAIM_FRAME_DROPSENSOR_INVALIDWhile the camera is trusted: every fifth dropped video packet in a session, at most once per 10 s. The next good frame reports SENSOR_VALID again.
SAIM_INTERNAL_ERRORSENSOR_INVALIDSAIM’s analysis failed to start or failed permanently for that camera.
SAIM_UNKNOWNSENSOR_INVALIDFallback when no specific cause is set.
SAIM_SENSOR_HEALTHYSENSOR_VALIDThe camera produced a sustained run of good frames.

Debounce. Each camera keeps a counter from 0 to counter_max (225). A frame scoring below 45 raises it, by more the worse the frame; a frame scoring above 55 lowers it; scores from 45 to 55 leave it unchanged. SENSOR_INVALID is reported when the counter reaches 225 and the frame is bad; SENSOR_VALID when it reaches 0 and the frame is good. With the default thresholds:

ChangeFrames neededAt 30 frames/s
Trusted to SENSOR_INVALID5 (score 0) to 225 (score 44)about 0.2 to 7.5 s
Untrusted to SENSOR_VALID25 (score 100) to 225 (score 56)about 0.8 to 7.5 s

The counter starts at 225 when a stream connects, so a camera becomes trusted only after a run of good frames.

Re-assertion. SAIM re-sends every camera’s current verdict, with its cause, every 30 s.

Camera failures

SituationBehaviour
A camera cannot be reached at startupSAIM retries 5 s after each failed attempt, for as long as it runs, and logs [<sensor>] RTSP connection failed, retrying every 5s on every 12th failed attempt: about once a minute when the camera refuses the connection at once, less often when the connection times out after 10 s. No new verdict is sent: the camera keeps its startup SENSOR_INVALID with cause SAIM_INIT, re-sent every 30 s. SEI marks its events unhealthy, but the SDMs do not count a SAIM_INIT camera as failed, so it never contributes to the all-sensors-failed latch. When the camera comes up, SAIM starts watching it and it becomes trusted through the normal debounce. Other cameras continue.
The stream stops mid-sessionAfter more than 5 s without packets, or on an RTCP BYE, the session ends. A trusted camera reports SAIM_STREAM_DISCONNECT at once ([<sensor>] SENSOR_INVALID handed to reporter (stream ended)). SAIM then reconnects every 5 s, for as long as it runs, and the camera recovers through the normal debounce.
A reconnection attempt failsA trusted camera reports SAIM_STREAM_DISCONNECT, if it has not already. SAIM keeps retrying every 5 s; it does not give up on a camera in ACTIVE.
The camera URL is malformedSAIM tries 6 times, 5 s apart, logs [<sensor>] RTSP: exceeded 5 connection retries (init failed), giving up, and stops that camera. A camera that was never trusted keeps SAIM_INIT.
The stream’s frame size differs from the size its baseline was learned atSAIM logs [<sensor>] profile was learned at <w>x<h> but the stream is <w>x<h>; re-run LEARN for this camera, then [<sensor>] FATAL: frame quality analyzer cannot start; shutting down this stream, reports SAIM_INTERNAL_ERROR and stops that camera. A size that does not fit the analysis grid always takes this path, because every baseline was learned at a size that fits. See Inputs and Baselines.
Analysis fails repeatedlyAfter 30 consecutive frame errors SAIM restarts that camera’s analysis, up to 5 times. After that it stops that camera and, if the camera was trusted, reports SAIM_INTERNAL_ERROR. The other cameras continue.
A baseline or GPU setup fails for one camera in ACTIVEThat camera’s thread ends without a new verdict, so it stays at SAIM_INIT. This includes a baseline from an earlier release (was learned on a 0x0 tile grid).
Every camera has stoppedSAIM exits 1 and unregisters. SEI raises SW_FAIL cause 15 and every SDM latches.

GPU selection

SAIM selects the GPU only through --gpu. Without it, camera i runs on GPU i modulo the number of visible GPUs. SAIM does not read CUDA_VISIBLE_DEVICES itself, but CUDA applies it: --gpu is an index among the GPUs that CUDA_VISIBLE_DEVICES leaves visible. An index out of range exits with Error: --gpu <N> is out of range (0-<max>).

LaunchWhat happens with --gpu <N>
launch_hoisa.sh --runtime host --gpu <N>SAIM gets --gpu <N>. CUDA_VISIBLE_DEVICES is passed on only if you set it.
launch_hoisa.sh --gpu <N> (Docker runtime) or the container option --gpu <N>The container launcher passes --gpu <N> and sets CUDA_VISIBLE_DEVICES=<N>. Only one GPU is then visible, as index 0, so any N other than 0 makes SAIM exit with the out-of-range error, and the container stops. Leave --gpu unset, or use 0.

Registration and supervision

ItemBehaviour
Client typeCLIENT_SAFETY_MONITOR on /run/nvpsf/nvpssd, in ACTIVE only. SAIM must run with the same primary group as the SEI daemon; see Shared process group.
RegistrationAfter the baselines load. Up to 6 attempts, 5 s apart, then Failed to register with PSS daemon after 6 attempts. Exiting. and exit 1.
HeartbeatEvery 5 s. A failed heartbeat is logged ([SAI] Heartbeat send failed); SAIM keeps running.
Single instanceSEI refuses a second ACTIVE SAIM (duplicate singleton client type rejected); the second process exits 1 after its retries. SEI accepts SENSOR_INVALID and SENSOR_VALID only from the registered SAIM.
Never registered or lostSW_FAIL cause 15. The never-registered budget is the same as for PCM: about 4.2 minutes with the shipped max_hb_failures = 50.

SAIM registers once, at startup. When the SEI daemon restarts, SAIM keeps running but does not register again, so the new daemon reports it as never registered when the budget runs out. Restart SAIM after restarting the SEI daemon.

Seeing the verdicts

SAIM writes to standard output and standard error, not to syslog. The launchers send both to /var/log/psf/saim.log; the container launcher also copies them to the container output (docker logs).

  • SAIM logs most SENSOR_INVALID hand-offs on standard error, as [<sensor>] SENSOR_INVALID handed to reporter (<reason>). It writes no line when a camera becomes SENSOR_VALID or when the frame score makes it SENSOR_INVALID.
  • Progress lines such as the mode banner and Notice: baseline … go to standard output, which SAIM does not flush. Under the launchers they can appear late or out of order relative to the standard-error lines.
  • For per-camera state changes, read the SDM log: each reference SDM logs safety-sensor health change pipelineID=<id> state=FAILED|RECOVERED cause=<n> for the cameras in its reviewed set. A camera at SAIM_INIT is not logged as failed.