Perception Container Monitor (PCM)

The Perception Container Monitor (PCM, nv_perception_monitor) judges whether the AI perception pipeline can be trusted. It checks the perception containers through the Docker Engine and the perception data on the Kafka topic mdx-frames, and reports one verdict for the whole pipeline, AI_PIPELINE_VALID or AI_PIPELINE_INVALID, to the SEI daemon.

PCM is part of the safety decision. It is required by default:

• The shipped nvpss.conf sets required_perception_monitor = 1. When a required PCM never registers, or is lost after registering, SEI raises a SW_FAIL with cause 17 and every SDM latches its safe state. See Registration and supervision.
• Every reference SDM latches its safe state on AI_PIPELINE_INVALID. See Effect on the safety decision.

Set required_perception_monitor = 0 only for a deployment that runs no ACTIVE PCM; see SEI configuration and Launchers.

PCM sends AI_PIPELINE_INVALID only for a fault it has seen in the containers or in the data. A setup problem, such as Docker or Kafka being unreachable or a missing baseline, is logged and turns the affected check off. See Faults and setup problems.

Modes

LEARN never switches to ACTIVE. Run them as separate invocations, with the same baseline location and the same decision inputs (--event-mapping and --sdm-config).

Command line

An empty environment variable counts as unset.

Checks

In ACTIVE, PCM runs up to two checks; the [monitors] section of pcm_monitor.cfg turns each on or off. Each check gives one result about once a second: good, bad or off. Off means the check has no result, because of a setup problem or because there is nothing to judge yet. Only a bad result can lead to AI_PIPELINE_INVALID.

Container health

About once a second PCM inspects each container named in the allow-list (monitor_container in [health_monitor]) by its exact name, through the Docker Engine API on /var/run/docker.sock. It does not list other containers and does not run the docker command.

A listed container is judged only after PCM has seen it in this run. A container that has never been present is not judged: PCM logs it once, startup_grace_ms after start, and checks it from the moment it appears.

The result of one poll is bad when any container is bad. Otherwise it is off while no listed container has been seen, or while any container is still starting within its grace. Otherwise it is good. A container that faulted stays bad while its health check is starting, and gets no new grace: it recovers only when it is running and healthy, or running without a health check.

When the Docker socket cannot be reached, or the Docker API returns an error or an incomplete reply for any container, the check is off for that poll. PCM logs the change once and resumes when Docker answers. A Docker failure does not restart a container’s starting grace. PCM also logs each container fault, recovery and late appearance once.

The shipped allow-list names the continuous producers of the reference warehouse deployment: vss-vios-nvstreamer, vss-vios-streamprocessing, vss-rtvi-cv, vss-behavior-analytics and kafka. List only services that must run all the time: a one-shot job that exits is a fault. Replace vss-vios-nvstreamer with the service that supplies your video.

When the perception containers run on another host, they never appear on the local Docker socket, so the check stays off. Turn container_health off there, and point the DVM at the remote broker with --kafka-broker.

Data Validation Monitor (DVM)

The DVM reads mdx-frames with its own Kafka consumer group, nv_perception_monitor_dvm, starting at the newest message. It checks each frame against the baseline of the camera that sent it, named by the frame’s sensorId. It counts frames per camera over 1-second windows and gives one result per window.

Each frame. A frame is bad when:

PCM compares frame timestamps with its own wall clock, so the perception host and the PCM host must keep their clocks in step.

Each window. The window result is bad when any of these holds:

• One camera’s bad frames reach bad_frames_percent of that camera’s frames in the window. The comparison is inclusive and rounds up: with 25, 8 bad frames out of 30 make the window bad, and 7 do not. Each camera is counted on its own, so healthy cameras cannot hide another camera’s bad frames.
• A camera that was already sending before the window began delivers fewer fresh frames than half its learned expected_fps. A fresh frame is within the freshness limits and has a later timestamp than that camera’s previous fresh frame; repeated or reordered timestamps do not count. This rule also catches every camera stopping at once while Kafka still answers. PCM logs a low frame rate line for each such window. A camera that keeps at least half its learned rate passes this rule; the freshness and decision-data checks still apply to its frames.
• Undecodable data was received in the window (see the next table).

Otherwise the window is good once at least one baseline camera has sent a frame in this run, and off before that. A baseline camera that has never sent a frame is not judged: PCM logs it once, startup_grace_ms after start, and checks it from its first frame. The window that holds a camera’s first frame is not rate-checked.

Kafka and decoding.

Decision data contract

The DVM checks the frame fields that the configured SDMs act on. PCM builds this contract at start from --event-mapping and --sdm-config:

1. From each SDM configuration it reads only the event bindings. An ATL configuration has exactly the six keys forklift_entered, forklift_exited, person_entered, person_exited, roi_violation and roi_cleared. A PXC configuration has pair_groups (default humanoid_person, forklift_person, transporter_person) and the <group>_normal, <group>_reduce and <group>_stop keys of each selected group. Every value is EVENT_0 to EVENT_23.
2. It selects the mapping rules whose output_event an SDM configuration binds. Without --sdm-config it selects every rule in the mapping and logs a warning.
3. It turns each selected rule into a contract entry:

PCM stops at start with exit status 2 and DVM decision contract: <reason> when:

Without --event-mapping, PCM logs a warning that starts with DVM: no current event mapping. LEARN then records no contract, and ACTIVE uses the contract stored in each baseline as it is, without checking it against the SDMs.

With --event-mapping, ACTIVE does not use a baseline record whose stored contract differs from the current one, and logs DVM: decision mapping changed for <camera>. Learn again whenever the event mapping or the SDM configurations change.

What PCM does not check

• Bounding boxes, detection confidence, object speed, and which object classes or how many objects are in the scene. bad_objects_percent is required in the configuration but has no effect.
• Tripwire events on mdx-events.
• A camera that has never sent a frame in this run. It is only logged, after startup_grace_ms.
• A container that has never been present in this run, or a container that is not in the allow-list.
• Consistency between redundant sensors, detection uncertainty, and model consistency.
• Image quality. SAIM checks each camera’s video; see Safety AI Monitor (SAIM).

Faults and setup problems

Verdict aggregation

An evaluator combines the latest result of each enabled check once a second. Each evaluation is a tick:

• A check that is off contributes nothing.
• A check that has not reported for longer than max_report_age_ms is stuck. Before its first report this is timed from start. PCM logs it, ignores its result, and counts no good ticks until it reports again. Being stuck never sends AI_PIPELINE_INVALID by itself.
• While the pipeline is INVALID, a check that stops contributing, because it goes off or stuck, owes a good result: good ticks do not resume until it reports good again. Losing sight of the pipeline does not prove that the fault cleared.

The pipeline starts in STARTING:

Counts are evaluator ticks, not exact elapsed time: the phases of the container poll, the DVM window and the tick add to the delay.

Sending.

• In STARTING PCM sends nothing.
• The first VALID or INVALID is always sent, even when SEI already holds that verdict. A restarted PCM therefore republishes its state: its first AI_PIPELINE_VALID clears an AI_PIPELINE_INVALID that a previous PCM left in SEI. A latched SDM still needs an operator release.
• A change is sent on the tick that makes it.
• An unchanged verdict is sent again verdict_resend_ms after the last successful send: 5000 ms in the shipped file and when the key is missing. A re-send never makes a new judgement.
• A failed send is logged and retried on the next tick. It does not move the re-send deadline.
• Each verdict is an AI_PIPELINE_INVALID or AI_PIPELINE_VALID event with cause 0 (CAUSE_NONE), pipeline ID 0 and confidence 1.0.

Configuration

ACTIVE reads pcm_monitor.cfg once at start, before it registers with SEI, from $PCM_MONITOR_CONFIG or /opt/nvidia/psf/bin/pcm_monitor.cfg. LEARN does not read it. Nothing is reloaded while PCM runs: edit the file and restart PCM.

The shipped file, without its comments:

[monitors]
container_health = true
data_validation = true
[fault_aggregator]
bad_ticks_to_invalid = 4
good_ticks_to_valid = 8
max_report_age_ms = 4000
initial_valid_ms = 4000
verdict_resend_ms = 5000
[data_validation]
bad_objects_percent = 25
bad_frames_percent = 25
startup_grace_ms = 10000
[health_monitor]
container_startup_grace_ms = 30000
monitor_container = vss-vios-nvstreamer
monitor_container = vss-vios-streamprocessing
monitor_container = vss-rtvi-cv
monitor_container = vss-behavior-analytics
monitor_container = kafka

Lines that start with # are comments. A comment after a value on the same line is not supported: it becomes part of the value.

The numeric ranges only prevent overflow; they are not safety limits. Choose the counts within the deployment’s fault-response and recovery budget. Very small values are accepted and behave literally: a max_report_age_ms below one second marks normal reporting as stuck, so PCM never sends AI_PIPELINE_VALID.

Baselines

The DVM needs a baseline that LEARN recorded from the running pipeline.

Location

PCM finds the baseline path in this order, in both modes:

1. --baseline-config <path>
2. PCM_BASELINE_CONFIG
3. /opt/nvidia/psf/bin/pcm_baseline/pcm_baseline.cfg

Only the directory part of the path is used, and the baseline file in it is always named pcm_baseline.cfg. /data/pcm/site.cfg selects /data/pcm. /data/pcm without a trailing slash selects /data. A name without / selects the working directory.

When PCM uses a directory other than /opt/nvidia/psf/bin, and that folder holds pcm_*_baseline.cfg or pcm_baseline.cfg files, ACTIVE logs that they are ignored.

What ACTIVE loads

1. When the directory holds pcm_baseline.cfg, ACTIVE uses only that file. When it is not a regular file or is invalid, data validation is off, and PCM falls back to no other file. PCM logs DVM: invalid or unreadable baseline camera set <path>.
2. Only when there is no pcm_baseline.cfg, ACTIVE reads every per-camera file pcm_<camera>_baseline.cfg and every template pcm_<camera>_baseline.cfg.default in the directory. A template is used only when the matching file without .default cannot be opened. A readable file that is malformed is skipped and is not replaced by its template.
3. With --event-mapping, ACTIVE drops every record whose contract differs from the current one.
4. When no camera is left, data validation is off.

A per-camera file without schema_version = 2 holds only known_sensor and expected_fps. It gives the unknown-camera and frame-rate checks only, and PCM logs a warning for it.

Reference baselines

The Debian package and the container image install three templates in /opt/nvidia/psf/bin/pcm_baseline/: pcm_Camera_baseline.cfg.default, pcm_Camera_01_baseline.cfg.default and pcm_Camera_02_baseline.cfg.default. They describe the three cameras Camera, Camera_01 and Camera_02 of the reference loading-dock dataset at about 29.9 frames per second, with max_frame_age_ms = 2000. Their decision contract is that of ATL zone 1 with the forklift–person PXC pair: proximity_event_mapping.pb.txt, atl_sdm_zone1.conf and pxc_sdm_forklift.conf. It has one ROI rule, roi-id-1 person restricted, and one pair, forklift person at 2 m.

• With other camera names, every frame comes from an unknown camera, and the DVM reports bad.
• With another decision contract given through --event-mapping, for example ATL alone or PXC alone, ACTIVE drops all three templates and runs with data validation off.
• For any other cameras or SDM profile, run LEARN. After a successful LEARN, ACTIVE reads only the new pcm_baseline.cfg; the templates stay on disk but are not read.

LEARN

LEARN observes mdx-frames for --learn-duration seconds and records, for each camera:

LEARN writes nothing, leaves the existing baseline unchanged, and exits 1 when:

• it is stopped before its duration ends (LEARN stopped before its <n> s duration; ...);
• it sees no frames, or it sees 10 consecutive undecodable messages at any time;
• a camera has fewer than 2 windows at or above half its median rate, or has more than one window, and more than one in 20 of its windows, below that;
• a camera ID contains a control character or =, or two camera IDs give the same file name once every character other than letters, digits, ., _ and - is replaced by _ and the name is cut to 128 characters;
• 25% or more of a camera’s frames fail the decision contract;
• the contract is set and the frames carry no timestamps, or only some frames carry usable timestamps;
• the 1st-percentile frame age is more than 2 s ahead of PCM’s clock, or the 99th-percentile age is above 14 750 ms;
• the contract has pair rules and no socialDistancing threshold is seen, or the median threshold is below a pair’s distance. PCM logs no reason for these two cases;
• the directory cannot be opened or flushed, or pcm_baseline.cfg exists and is not a regular file.

The write is atomic. LEARN writes every camera into a hidden temporary file in the same directory, flushes it, and renames it over pcm_baseline.cfg. Until the rename, ACTIVE sees the previous file. LEARN never writes or deletes per-camera files or templates. The baseline directory must exist, must be writable by PCM, and must not be a symbolic link.

Run LEARN while the pipeline is healthy. Learn again after you change cameras, the scene, the event mapping or the SDM configurations.

File format

LEARN writes pcm_baseline.cfg like this. Do not edit it by hand.

PCM_BASELINE_SET_V1
cameras = 3
# PCM decision-data baseline; generated by --mode LEARN
[dvm_baseline]
schema_version = 2
known_sensor = Camera
expected_fps = 29.910269192422732
max_frame_age_ms = 2000
decision_contract = 1
proximity_required = 1
proximity_threshold = 4
pair_rule = forklift person 2
roi_rule = roi-id-1 person restricted
[dvm_baseline]
...

ACTIVE rejects the whole file when the header or the camera count is wrong, any record is invalid or repeats a camera, or the records hold different contracts.

Registration and supervision

Effect on the safety decision

SEI keeps one AI-pipeline trust flag for the whole pipeline. AI_PIPELINE_INVALID sets it and AI_PIPELINE_VALID clears it; it is not cleared when PCM is lost. While it is set, SEI marks every event untrusted (isTrustedSource = false) and re-sends AI_PIPELINE_INVALID with every status no-op. SEI accepts these verdicts only from CLIENT_PERCEPTION_MONITOR with a fresh heartbeat. See Safety Event Integrator (SEI).

Every reference SDM, on the CCPLEX and on the FSI, handles the verdict the same way:

• AI_PIPELINE_INVALID, unless STALE, latches the safe state and sends the safe hold at once. The SDM logs ... AI pipeline reported INVALID; entering safe hold.
• Events marked untrusted are logged and not used for decisions.
• AI_PIPELINE_VALID clears the condition only. A release request is denied while the condition is active, and the latch stays until an operator releases it. See Safe-state model.
• An SDM starts with the condition clear and relearns a standing AI_PIPELINE_INVALID from the SEI’s re-sends within one status interval.

Launchers

Both launchers run PCM; their options are in Launcher. From PCM’s side:

• --pcm-mode learn|active|skip selects PCM’s mode, independently of SAIM. When the launcher starts the SEI daemon, required_perception_monitor must be 1 for active and 0 for learn or skip, or the launcher stops before it starts anything.
• Under launch_hoisa.sh with the Docker runtime, PCM runs inside the Safety Core container, with the host’s Docker socket mounted. With --runtime host it runs on the host, as root in ACTIVE.
• The launchers pass --baseline-config <dir>/pcm_baseline.cfg, set PCM_MONITOR_CONFIG, add --kafka-broker when one is given. For the ATL and PXC application combinations they also add --event-mapping and, when the launcher reads SDM configurations, one --sdm-config for each, in both modes.
• Under the Docker runtime, ACTIVE PCM reads the configuration packaged in the image unless you pass --pcm-config <file> or set PCM_MONITOR_CONFIG. The launcher then mounts that file into the container read-only.
• PCM writes to /var/log/psf/pcm.log. When ACTIVE PCM exits, the launcher stops the Safety Core.

Logs

Every PCM line has the form [YYYY-MM-DD HH:MM:SS.mmm] [PCM][<level>] <message>, in local time, with level INFO, WARNING or ERR. INFO goes to standard output; WARNING and ERR go to standard error. DVM messages start with DVM: . Lines from the Kafka and SEI client libraries keep their own format. In the DVM warnings about unknown cameras and decision-data faults, every character of a camera ID other than letters, digits, ., _ and - is printed as _.

To see the latest verdict:

grep -E 'AI_PIPELINE_(VALID|INVALID)' /var/log/psf/pcm.log | tail -n 1