Skip to navigation

Integration Guide

View as Markdown

The Safety Core turns perception output into a safety decision and sends that decision to a PLC. It checks every handoff on the way, watches the health of its own inputs, and puts every decision maker into a latched safe state when it can no longer trust what it receives.

This guide is for integrators: people who connect their own perception, write their own decision maker or event source, or run the Safety Core without the reference launcher. To run the reference applications, use the Deployment Guide.

Safety Core overview: perception metadata, camera streams and perception containers feed the MDX client, SAIM and PCM; they report to the SEI daemon, which feeds the NvPSD Gateway, which feeds the SDMs on the CCPLEX or, through fsicom-agent, on the FSI; the SDMs command the PLC and accept a validated manual release from it

Components

The components, their binaries and their roles are listed in Safety Core Components, with a page and an architecture diagram for each.

Paths, package names, binaries and log lines use older short names: pss for the SEI, psd for the gateway and decision side, psb for the SBB, and psf for the Safety Core as a whole. For example, the gateway logs as PSD-Gateway and the SEI’s audit records start with PSS_.

Data path

One safety event travels through these stages:

#StageInputWhat happensOutput
1Perception (SIPP)Camera streamsDetection, tracking and analytics.Kafka mdx-events (Behavior messages: tripwire and ROI alerts) and mdx-frames (FrameMessage messages: per-frame objects, ROI and proximity).
2MDX clientKafka messagesDecodes them, matches each against the rules of the event-mapping file, and resolves the camera name to a pipeline id.A SafetyEvent (EVENT_0 to EVENT_23) reported to the SEI.
3Trust inputsCamera streams; perception containers and mdx-framesSAIM judges each camera’s input. PCM judges the perception pipeline as a whole.SENSOR_INVALID or SENSOR_VALID per camera; AI_PIPELINE_INVALID or AI_PIPELINE_VALID for the whole pipeline.
4SEI daemonEvents and verdictsChecks each report, stamps each event with the standing verdicts (isHealthy, isTrustedSource), marks old events STALE, optionally fuses detections of one object from several cameras, sets the severity, and supervises every source. Sends a status no-op with a presence report at a fixed interval.A DecisionRequest of up to 8 entries, with the SEI’s mode (NORMAL, DEGRADED, ERROR).
5NvPSD GatewayDecision requestsChecks the SEI’s request numbering, turns its own link faults into SW_FAIL records, filters each request to each SDM’s subscription, and wraps it in a numbered envelope per SDM.One DRPG envelope per SDM.
6SDMEnvelopesValidates the envelope, its CRC and its sequence, applies the verdicts and fault causes, decides, and latches the safe state on any fault.A 64-byte command to the PLC.
7PLC or command receiverCommandsActs on the command and acknowledges it. Sends a release request when an operator confirms that the area is safe.Acknowledgements; release requests.

How the perception messages map to safety events is described on Event Mapping Configuration. What makes an SDM latch, and how it is released, is described in the safe-state model.

The SEI daemon, the gateway and the MDX client write their records through the SBB to syslog. The SBB is not in the data path; see Logs for where each log lands.

Where each process runs

Platforms

PlatformWhere the Safety Core runs
x86-64, Ubuntu with an NVIDIA GPUEvery process runs on the CPU under Linux. SAIM uses the GPU.
IGX Thor (aarch64), SDM on the CCPLEXEvery process runs on the CCPLEX under Linux for Tegra. SAIM uses the PVA by default.
IGX Thor (aarch64), SDM on the FSIThe SDM runs as firmware on the FSI’s real-time OS. fsicom-agent runs on the CCPLEX in its place; everything else is as above. ATL and PXC only, 2D only.

Container or host

The Safety Core can be started in three ways. The container’s options are on Container options, and the launcher’s on launch_hoisa.sh.

ProcessPlain docker run of the imagelaunch_hoisa.sh --runtime docker (default)launch_hoisa.sh --runtime host
nvpss_daemon, nvpsd_gateway, mdx_clientContainerContainer nv-psfHost, as root
safety_monitor (unless --saim-mode skip)ContainerContainerHost
nv_perception_monitor (unless --pcm-mode skip)ContainerContainerHost
CCPLEX SDMsContainer; on the host only with --sdm-externalHost, as root with the container user’s groupHost, as root
fsicom-agent (SDM on the FSI)Host, started by you, with --sdm-on-fsiHost, as root with the container user’s groupHost, as root
Command receiver or PLCNot startedNot started by the launcherNot started by the launcher

Under --runtime docker the launcher starts the SDMs, or fsicom-agent, on the host as root, with the primary group of the container user nvidia, so the NvPSD Gateway in the container accepts them. See Shared process group.

With a plain docker run, every Safety Core process except fsicom-agent runs in the container by default, so no Safety Core link crosses the container boundary. The container needs:

  • Docker socket. Mount the host’s /var/run/docker.sock, so PCM can check the perception containers from inside the container.
  • Network. The container uses the host network (--network=host), so the SDM-to-PLC UDP link and the Kafka connection behave as on the host.
  • Files. Mount each configuration file over the path the container reads it from. The monitor modes must match required_safety_monitor and required_perception_monitor in the nvpss.conf the container reads.

A Safety Core process on the host that connects to the container, such as fsicom-agent or an SDM next to a container started with --sdm-external, needs two things more:

  • Sockets. All Safety Core sockets live in /run/nvpsf. Bind-mount that host directory into the container (-v /run/nvpsf:/run/nvpsf), so the process on the host and the processes in the container reach the same sockets.
  • One group. The process must run with the container user’s primary group. See Shared process group.

fsicom-agent always runs on the host: it talks to the FSI through NvFsiCom, which needs the FSI channel in the host’s device tree and the NvFsiCom daemon running on the host.

2D and 3D deployments

deploymentMode in nvpss.conf is mandatory and states the topology. It is not inferred from which files are present.

AspectdeploymentMode = 2DdeploymentMode = 3D
Event identityEach camera reports its own events.Cameras are calibrated into one BEV group, and events are reported under the group.
nvpss.confFusion may be on or off (enableFusion).Requires enableFusion = false and sensorGroupConfig, the sensor group file. The daemon refuses to start otherwise.
mdx_client identity file--sensor-config sensor_config.conf (the cameras)--sensor-pipelines-config sensor_pipelines_config.conf (the group only)
SDM reviewed sensor set (sensor_pipelines_config.conf)The camerasThe group only
SAIMWatches every camera in sensor_config.conf.The same.
Sensor verdicts the SDMs seePer camera.Per group. The group is untrusted only when every member camera is invalid and none is still initialising; one healthy member keeps it trusted. SAIM cannot report the group directly.
SDM on the FSISupportedNot supported

The file formats are on Sensor files. The keys are on SEI Configuration.

Transports

Every link inside the Safety Core is an AF_UNIX SOCK_SEQPACKET socket in /run/nvpsf. None of them can be switched to UDP or TCP. Only the links to the outside world use the network.

#LinkTransportAddressServerClientCredential check
1Perception to mdx_clientKafkaTopics mdx-events, mdx-frames. Broker from --broker, else MDX_MSGBUS_BROKERS, else localhost:9092.Kafka brokermdx_client—
2Perception to PCMKafka; Docker Engine APITopic mdx-frames, with its own consumer; /var/run/docker.sock, mounted into the container under the Docker runtimeKafka broker; Dockernv_perception_monitor—
3Cameras to SAIMRTSPThe rtspUrl of each camera in sensor_config.confCamera or stream serversafety_monitor—
4Sources to the SEI (SEI RPC)AF_UNIX SOCK_SEQPACKET/run/nvpsf/nvpssd, mode 0660nvpss_daemonmdx_client, safety_monitor, nv_perception_monitor, nvpsd_gateway, your own sourcesServer checks the client
5SEI to gatewayAF_UNIX SOCK_SEQPACKET/run/nvpsf/nvpssd_to_psd, mode 0660nvpss_daemonnvpsd_gatewayBoth ends
6Gateway to SDMsAF_UNIX SOCK_SEQPACKET/run/nvpsf/nvpsd_gateway, mode 0660nvpsd_gatewayEach SDM, or fsicom-agentBoth ends
7SDM to PLCUDP, 64-byte packetsThe SDM sends from an ephemeral port to --cmd_rx_ip:--cmd_rx_port; the receiver listens on -p (default 12345).PLC or command receiverSDMSource address, identifier and CRC
8fsicom-agent to the FSI SDMNvFsiCom, channel 6Device-tree node compatible with nvidia,tegra-fsicom-bufdemo-10k———
9fsicom-agent to PLCUDP, 64-byte packets--ip:--portPLC or command receiverfsicom-agentSource address, identifier and CRC
10Display relays (optional)UDPATL command receiver --vst_ip:--vst_port; proximity_sdm --halo_ip:--halo_portVSTReceiver or SDM—

Notes on the AF_UNIX links:

  • Directory. /run/nvpsf must exist before any server starts; no binary creates it. The runtime package creates it with mode 1777 at install and on every boot, and the launchers check it; see Install.
  • Socket mode. Each server sets its socket to 0660 after binding. The parent directory protects nothing, so access control is the credential check.
  • Second instance. A server that finds a live server already listening on its path refuses to start and prints nvpsf_transport: bind(<path>) refused -- a LIVE server is already listening on this path. A stale socket left by a crash is removed and reused.
  • Buffers. The ends of link 6 request socket buffers for 516 decision packets. The kernel limits must allow them; see Kernel socket buffers.

Shared process group

Every Safety Core process must run with the same primary group. On every AF_UNIX link, the server reads each client’s credentials with SO_PEERCRED and compares the client’s group with its own getgid(). On links 5 and 6 the client also checks the server the same way. The peer’s primary group must match exactly; having the group as a supplementary group is not enough. The rule covers nvpss_daemon, nvpsd_gateway, mdx_client, safety_monitor, nv_perception_monitor, every SDM, fsicom-agent, and every process you add.

There is no dedicated Safety Core group. Each process requires its own primary group of its peers, so the rule holds when every process runs with the same primary group, whichever it is.

DeploymentPrimary group of the Safety Core processesMeets the rule
Plain docker run of the image, every process in the containerThe group of the container user nvidia. The entrypoint runs the container launcher as nvidia, and the launcher keeps that group when it starts SAIM and PCM with extra device or Docker socket groupsYes
launch_hoisa.sh --runtime hostRoot’s group, 0: the launcher starts every process that connects to another one with sudo -nYes
Every process started by hand in one root shell, as in Running without Docker0Yes
launch_hoisa.sh --runtime docker, control-plane runThe container user’s group. The launcher reads it from the running container and starts the SDMs or fsicom-agent on the host as root, with that group as their primary groupYes

The container user’s group ID is assigned when the image is built. Read it from the image, or from a running container:

docker run --rm --entrypoint id nvcr.io/nvidia/halos-outside-in/outside-in-safety:nv-psf-halos-1.4-6b92731 -g nvidia
docker exec <container-name> id -g nvidia

For a process you start yourself

Start the process with the primary group of the processes it connects to. Next to processes that run as root on the host, start it as root. Next to the container, start it as root with the container user’s group ID as its primary group, keeping root’s user ID and supplementary groups:

sudo setpriv --regid <container-gid> --keep-groups -- <command>

<container-gid> is the number printed by the id command above. The container must also mount the host’s /run/nvpsf, so that the process on the host sees the container’s sockets. The SDM on FSI procedure starts fsicom-agent this way, and launch_hoisa.sh --runtime docker starts its SDMs and fsicom-agent the same way. Docker must not use user-namespace remapping or rootless mode: the host then sees the container’s group IDs shifted, and no host process can share them.

Host accounts with the container’s group

When a process on the host connects to the container, /run/nvpsf is shared with the host, and the container user’s group ID is the only credential the sockets check. A host account that can run with that group ID can then connect to the Safety Core sockets, or stand in for a Safety Core server whose socket is not bound, for example before the NvPSD Gateway starts or after it exits. That is an account whose primary group has that ID, or a member of a host group with that ID, who can switch to it with newgrp or sg. An SDM or fsicom-agent on the host would then accept such a process as its gateway. A host account with the container user’s user ID owns the socket files on the host and can delete them, so that processes connecting or reconnecting afterwards cannot reach the Safety Core, and the decision makers then hold their safe state.

launch_hoisa.sh --runtime docker prints a warning at every control-plane start when either ID is used on the host; see Host processes and the container’s group. To check a host yourself:

getent group <container-gid>
getent passwd | awk -F: '$4 == <container-gid>'
getent passwd <container-uid>

<container-uid> is printed by docker exec <container-name> id -u nvidia. Each command prints nothing when no host group or account uses the ID. Keep such accounts off the host, or run every Safety Core process in the container without the /run/nvpsf mount, so that no socket is shared with the host.

What a mismatch looks like

A mismatched process is refused. This is the safe direction: it cannot register, so an SDM holds its safe output and latches on lost freshness and gateway heartbeat, and a required monitor that never registers is reported lost to every SDM. The refusing end prints both group ids to standard error:

nvpsf_transport: peer GID <peer-gid> does not match the required PSF group <own-gid>. Every PSF process must share one GID; check the container UID/GID mapping across the boundary this link crosses.

followed by one of:

Printed byLine
The server (nvpss_daemon, nvpsd_gateway)nvpsf_transport: rejected connection, peer credential is not in the allowed group
A transport client (an SDM, fsicom-agent, or the gateway on link 5)nvpsf_transport: server credential rejected; refusing to treat the connection as trusted

The gateway logs the group it requires at start-up, in a syslog record whose data reads PSD-Gateway: peer credential allowlistrequiredPrimaryGid=<gid>; search for requiredPrimaryGid=. To compare it with the running processes:

ps -eo pid,egid,args | grep -E 'nvpss_daemon|nvpsd_gateway|mdx_client|safety_monitor|nv_perception_monitor|_sdm|fsicom-agent'

Integration Guide pages

PageContents
InstallDebian packages, install paths, runtime libraries, kernel settings, running every process without Docker, building the reference applications.
Writing a decision makerThe contract a custom SDM must meet.
Safe-state modelWhat latches an SDM, what clears each cause, and how the latch is released.
Custom event sourcesAdding your own event source or monitor through the SEI client library.
SAIM distortion injectionInjecting camera faults into a test deployment to check SAIM and the decision makers.

The Reference section holds the full option and key lists: Launcher, SEI Configuration, Sensor files, Event Mapping Configuration and Causes and Records.