Custom Deployment
Start from a working Quick Deployment. This page tells you which file to change for each kind of change, and what else the change implies. The formats and every key are in the Reference: Sensor files, SEI Configuration, Event Mapping Configuration and Container options.
Site files
The examples keep the site’s files in ~/hoisa-site and mount each of them into the container.
Start from the packaged files and the camera file of the Quick Deployment:
Expected: atl_sdm.conf, nvpss.conf, pcm_monitor.cfg, sensor_config.conf and
sensor_pipelines_config.conf.
Configuration map
The container reads each file at a fixed path. The start command
mounts every site file over that path, so you change a file in ~/hoisa-site and start the
container again.
Configuration is read at start-up. After any change, stop and remove the container as in
Stop and run the start command again. A container started
again with docker restart keeps the options it was created with.
Start with your site files
These two commands replace the learn and start steps of the Quick Deployment. Every procedure on this page uses them.
-
Learn the baselines. Point the cameras at the normal, unobstructed scene. The container runs in the foreground and is removed when it exits.
--learn-durationaccepts 1 to 3600 seconds and defaults to 300.Expected:
SAIM completed LEARN successfully,PCM completed LEARN successfully, thenAll selected monitors completed LEARN successfully. -
Start the Safety Core.
Expected:
Required sources: safety_monitor=1 perception_monitor=1 (from /opt/nvidia/psf/bin/nvpss.conf)andAll processes started. Waiting for shutdown signal.... Then start the command receiver as in Start the command receiver and run Check that it is working.
Your own cameras
These steps are for 2D perception, where each camera reports its own events.
-
Write the camera file. One line per camera in
~/hoisa-site/sensor_config.conf:pipelineId, sensorName, rtspUrl. Use pipeline IDs 1 to 8, each once.sensorNamemust be exactly the VST sensor name, which perception puts in each event.rtspUrlmust be the VST proxy URL of that camera (rtsp://<host>:<port>/live/<id>), as listed bycurl -s http://127.0.0.1:30888/vst/api/v1/sensor/streams.Expected: one line per camera, with no duplicate ID or name. Check every URL with the
ffprobeloop from Check that each camera stream plays. -
List the safety-relevant cameras. The decision makers read this list, the reviewed sensor set, from
~/hoisa-site/sensor_pipelines_config.conf, with two fields per line:pipelineId, sensorName. Edit it so that it lists the cameras the site actually runs, with the same IDs and names assensor_config.conf. A camera listed but never reported can never fail, so the all-cameras-failed safe state can never trigger.Expected: one line per safety-relevant camera.
-
Set the pipeline count. In
~/hoisa-site/nvpss.conf, setmaxPipelinesto at least the highest pipeline ID.Expected:
maxPipelines=<n>, where<n>is at least your highest pipeline ID. -
Learn new baselines. Follow Baselines.
-
Match the calibration to the event mapping. The forklift safety function needs a tripwire and an ROI whose IDs equal the
rule_idvalues in the event mapping (tripwire-id-1androi-id-1by default), and the ROI must listPersoninrestrictedObjectTypes. Check it as in Prerequisites.
The decision maker logs the reviewed set it loaded:
grep 'reviewed sensor set' /var/log/psf/atl_sdm.log prints the IDs and names. Events from a
camera that is not in sensor_config.conf are dropped, with
Dropping event: sensor not in sensor_config, so it has no pipelineID: <name> in
docker logs nv-psf.
3D perception. In 3D, events arrive under the name of a camera group, so nvpss.conf needs
deploymentMode = 3D, enableFusion = false and sensorGroupConfig, and
sensor_pipelines_config.conf names the group instead of the cameras. The container then also
needs the sensor group file: add a read-only mount of it at the path that sensorGroupConfig
names. The file formats are in Sensor files. The
decision maker on the FSI does not support 3D.
Baselines
Both monitors compare what they see with baselines learned on the normal scene.
- SAIM uses, for each camera,
<sensorName>_baseline.cfgfrom/opt/nvidia/psf/bin/saim_baseline, and falls back to<sensorName>_baseline.cfg.default. The package ships.defaultbaselines only for the demo cameras (Camera,Camera_01andCamera_02). With neither file, SAIM exits at start-up withError: baseline file '<name>_baseline.cfg' (or '<name>_baseline.cfg.default') not found for sensor '<name>'. - PCM uses one file,
/opt/nvidia/psf/bin/pcm_baseline/pcm_baseline.cfg, for all cameras. It also records the event bindings of the decision makers, taken from--appand--sdm-config. An active run whose bindings differ does not use it, and PCM then runs without its data check. The shipped reference files fit only the demo cameras with--app atl,pxc; see PCM baselines.
Learn both again after you move, rename, add or remove a camera, or change the scene. Learn PCM
again after you change --app, an SDM configuration, --event-mapping, or where the decision
maker runs (CCPLEX or FSI); for that, use --saim-mode skip in the learn command, and leave out
--sensor-config and the GPU options.
-
Learn. Run the learn command of Start with your site files, with the same
--app,--sdm-configand--event-mappingthat the active run will use.Expected:
All selected monitors completed LEARN successfully. One<sensorName>_baseline.cfgper camera is in/opt/nvidia/psf/bin/saim_baseline/, and/var/log/psf/pcm.loghas a line ending inDVM: baseline: published <n> camera(s) in /opt/nvidia/psf/bin/pcm_baseline/pcm_baseline.cfg; bbox/confidence/occupancy are not fault baselines. To learn one monitor only, set the other toskip. -
Start the active run with the same baseline directory mounts. A new PCM baseline replaces the whole camera set, so cameras you no longer run drop out of it.
-
Check that PCM uses the baseline.
Expected:
data_validation=on (<n> baseline camera(s)), where<n>is your number of cameras.data_validation=off (no baseline)means no usable baseline remained: it is missing, unreadable or corrupt, or the learn run used other--app,--sdm-configor--event-mappingvalues. The warnings logged just before thechecks:line give the cause. Then confirm that the checks report results, as in Check that it is working.
Required monitors
Every start selects both monitors: --saim-mode for SAIM, which reports each camera valid or
invalid, and --pcm-mode for PCM, which reports the AI pipeline valid or invalid. Each takes
learn, active or skip. Three mandatory keys in nvpss.conf declare the safety sources the
SEI daemon requires, and the container checks two of them against the modes:
A mismatch stops the container before anything starts, with
Error: --pcm-mode skip conflicts with required_perception_monitor = 1 in /opt/nvidia/psf/bin/nvpss.conf.
in docker logs nv-psf. The container never edits the file. A learn run with no monitor in
active does not read it and starts no decision maker.
What happens at run time:
- A required source that has not registered within the heartbeat budget after the SEI daemon
starts (about 4 minutes at the shipped
max_hb_failures = 50) is reported lost withreason=never_registered. Every decision maker then latches its safe state and refuses release until that source is back. - A source that registered and is then lost, because it crashed, stopped sending heartbeats or disconnected, latches every decision maker in the same way, whether it is required or not.
- A missing, repeated or out-of-range key stops both the container and the SEI daemon at start-up.
Running without a monitor. Skipping a monitor removes its protection. To run without PCM, set
required_perception_monitor = 0 in ~/hoisa-site/nvpss.conf and use --pcm-mode skip in the
start command. For SAIM, use required_safety_monitor = 0 and --saim-mode skip.
PCM container checks. PCM inspects each container listed as monitor_container = <name> in
its configuration, through the host’s Docker socket, which the start command mounts into the
container. The shipped list names the five containers that carry the video and the perception
output in the VSS Warehouse pipeline. Once PCM has seen a listed container, that container
stopping (with any exit code), pausing, restarting, turning unhealthy or disappearing makes PCM
report AI_PIPELINE_INVALID. A listed container that never appears is only reported in pcm.log,
and with no valid entry the container check is off. To change the list, edit
~/hoisa-site/pcm_monitor.cfg:
Expected: five lines, vss-vios-nvstreamer, vss-vios-streamprocessing, vss-rtvi-cv,
vss-behavior-analytics and kafka. Edit them to the names on this host.
When perception runs on another host, PCM cannot see its containers: in
~/hoisa-site/pcm_monitor.cfg, set container_health = false in the [monitors] section, keep
data_validation = true, and add --broker <host>:<port> to the container options of the learn
and start commands. The other keys of the file are on
Perception Monitor.
Decision maker configuration and event mapping
Each decision maker reads a configuration that names the event codes it acts on. The package
ships samples; copy one into ~/hoisa-site, review it, mount it into the container and name its
container path with --sdm-config.
An unknown, repeated or missing key stops the decision maker at start-up, with the reason in
docker logs nv-psf, and the container then stops.
The MDX client turns Kafka messages into those codes with an event mapping. The container picks
/opt/nvidia/psf/apps/atl/event_mapping_atl.pb.txt for atl and atl,atl, and
/opt/nvidia/psf/apps/proximity/proximity_event_mapping.pb.txt for every --app value that
includes pxc. To use your own, mount it read-only and pass its container path with
--event-mapping, in the learn and the start command; its output_event values must match the
decision maker configurations.
Command receiver endpoint and ports
--cmd_rx_ipis the IPv4 address of the command receiver or PLC, and--cmd_rx_portits UDP port, one port per decision maker in--apporder. The decision maker accepts acknowledgements and release requests only from that address and port.- A command receiver on another machine must be told where the decision maker runs:
atl_sdm_cmd_receiver -p <port> --sdm_ip <safety-core-host-ip>. It drops packets from any other sender withDropped packet from unexpected SDM sender. - Keep the receiver’s
--max_hb_failuresat its default of10. The receiver then stops about 55 seconds after the last decision maker heartbeat. A larger value delays that fault. - To draw the decision on video, both receivers forward commands to VST with
--vst_ip <vst-ip> --vst_port <port>; the port must equal VST’shalo_safety_udp_port. - The Proximity Monitoring halo around forklifts is drawn when
proximity_sdmsends its commands to a VST halo listener too. The container has no option for this; onlylaunch_hoisa.sh --halo-ipsets it.
Running more than one decision maker
Pass two applications to --app, with one configuration for each, in the same order. Learn the
baselines with that --app and both configurations, then start the active run with one receiver
port for each decision maker, again in the same order. This example runs the forklift safety
function and Proximity Monitoring together:
-
Copy the second configuration and create its log file.
Expected: a line starting with
-rw-rw-rw-. -
Learn with both configurations. A learning run needs no receiver ports.
Expected:
All selected monitors completed LEARN successfully. -
Start the Safety Core with the same
--appand configurations, and one port for each decision maker.Expected:
All processes started. Waiting for shutdown signal.... -
Start the forklift safety function receiver in its own terminal, with its own log file. Both receivers write to
/tmp/cmdrx.logby default, so without--log-filetheir records mix.Expected:
ATL UDP Command Receiver listening on port 12350. -
Start the Proximity Monitoring receiver in another terminal, with its own log file.
Expected: a line that starts with
Proximity UDP Command Receiver listening on port 12351.
Decision maker on the host
The decision maker can run on the host, outside the container, for example to run an SDM you
built yourself. The SEI daemon, the NvPSD Gateway, the MDX client and the monitors stay in the
container. The SDM connects to the gateway in the container through the shared socket directory
/run/nvpsf, and the gateway admits it only when it runs with the container user’s primary group
ID (see Shared process group). These steps run the forklift
safety function with the files, baselines and command receiver of
Quick Deployment, which must have run up to and including
Learn the baselines. You need three terminals.
launch_hoisa.sh --runtime docker starts the same layout for you; see
Host processes and the container’s group.
-
Check the socket directory. The host directory
/run/nvpsfis mounted into the container in the next step, so the gateway’s socket is visible on the host.Expected: a line starting with
drwxrwxrwt. -
Start the Safety Core without the decision maker. In terminal 1, start the container with
--sdm-externaland the/run/nvpsfmount.--sdm-configis passed for PCM only, which checks the decision maker’s event bindings.Expected:
SDMs run outside this container; skipping local SDM startupandAll processes started. Waiting for shutdown signal.... -
Read the container user’s group ID.
Expected: one number other than
0. That number is<container-gid>. -
Start the decision maker. In terminal 3, start
atl_sdmas root with the container user’s group ID as its primary group. It sends its commands to the command receiver at127.0.0.1:12350and writes its log to/var/log/psf/atl_sdm.log.Leave it running.
-
Start the command receiver in terminal 2, as in Start the command receiver.
-
Check that the gateway admitted the decision maker.
Expected:
0; oneatl_sdmline with user ID0and both group IDs equal to<container-gid>; andSDM: sent registration (7 event types) to gateway. Then run the checks in Check that it is working.
To stop, press Ctrl-C in terminal 3, then stop the container and the receiver as in
Stop. Started as root without setpriv, the decision maker
runs with group ID 0 and is refused: the container output shows
nvpsf_transport: peer GID 0 does not match the required PSF group <container-gid>., and the
decision maker holds its safe state.
SDM on FSI
On an IGX Thor the decision maker can run on the Functional Safety Island (FSI) instead of the
CCPLEX. The NvPSD Gateway, SEI daemon, MDX client and monitors stay in the container, and
fsicom-agent, which you start on the host, relays decisions between the gateway, the FSI and the
command receiver. fsicom-agent connects to the gateway in the container through the shared
socket directory /run/nvpsf, and the gateway admits it only when it runs with the container
user’s primary group ID (see Shared process group). This mode
runs one decision maker, the forklift safety function or Proximity Monitoring, on 2D perception.
The FSI firmware has its event codes and its reviewed cameras (Camera, Camera_01 and
Camera_02) built in, so --sdm-config and sensor_pipelines_config.conf do not apply.
-
Flash the platform. Flash the IGX Thor BSP and the Safety Extension Package (SEP), as described in the IGX Safety Extension Package documentation.
-
Install the FSI package on the IGX Thor. It provides
/opt/nvidia/psf/bin/fsicom-agentand the firmware/opt/nvidia/psf/etc/fsi-fw/<app-dir>/fsi-ffw-t264.bin. The package needs a separate sign-up at the early-access page.Expected: both paths are listed.
-
Flash the application firmware. Copy
fsi-ffw-t264.binfromfsi-fw/atl/orfsi-fw/proximity/to the host that flashes the IGX Thor, and flash it in place of the default FSI firmware from the SEP, following the SEP documentation. The firmware must match the--appyou start. -
Reset the board. The FSI runs the new firmware only after a reset or reboot. Do this before the next step.
-
Raise the socket buffers.
fsicom-agentneedsnet.core.rmem_maxandnet.core.wmem_maxof at least5283840, more than the VSS kernel settings give.Expected: both settings printed back. The file name sorts after the VSS settings file, so the higher value also holds after a reboot.
-
Start the platform
nvFsiComdaemon. Nothing in the Safety Core starts it, andfsicom-agentcannot reach the FSI without it.Expected: one
nvFsiComprocess. -
Learn the PCM baseline for this mode. Without an SDM configuration, PCM checks every rule of the application’s event mapping, so it needs a baseline learned the same way.
Expected:
PCM completed LEARN successfully, thenAll selected monitors completed LEARN successfully. -
Check the socket directory and read the container user’s group ID. The host directory
/run/nvpsfis mounted into the container in the next step, so the gateway’s socket is visible tofsicom-agenton the host.Expected: a line starting with
drwxrwxrwt, then one number. That number is<container-gid>. -
Start the Safety Core with the decision maker on the FSI. Use
--sdm-on-fsi, no--sdm-config, and the same<app>as the firmware.Expected:
SDMs run outside this container; skipping local SDM startupandAll processes started. Waiting for shutdown signal.... -
Start
fsicom-agentin a third terminal, as root with the container user’s group ID as its primary group. It relays to the command receiver at127.0.0.1:12350.Expected:
[INIT] ✓ FSI communication initialized,[INFO] CMD identifier: <cmd-identifier>, a line starting[GW] Connected to gateway, and[HB] First keepalive received. Leave it running; it prints to this terminal only. -
Start the command receiver as in Start the command receiver. For Proximity Monitoring use
/opt/nvidia/psf/apps/proximity/proximity_sdm_cmd_receiver. -
Check that the gateway admitted the agent.
Expected:
0.
To stop, press Ctrl-C in the fsicom-agent terminal, then stop the container and the receiver
as in Stop. In this mode the decision maker on the FSI logs
to the FSI UART; use tcu_muxer to find its device node. The receiver logs
UTC epoch: 0.000000 for every command, because the FSI decision maker leaves the packet time
stamp at zero. The fsicom-agent options are listed in
fsicom-agent options.
Proximity Monitoring
Run Proximity Monitoring like the forklift safety function, with its own configuration, log file and receiver.
-
Copy the configuration and create its log file.
Expected: a line starting with
-rw-rw-rw-. -
Learn the baselines with
--app pxcand this configuration.Expected:
All selected monitors completed LEARN successfully. -
Start the Safety Core.
Expected:
All processes started. Waiting for shutdown signal.... -
Start the receiver in its own terminal.
Expected: a line that starts with
Proximity UDP Command Receiver listening on port 12350.
Its log is /var/log/psf/pxc_sdm.log. Perception must report proximity between the configured
machine and Person on mdx-frames; see
Proximity Monitoring.