NVIDIA UFM Enterprise User Manual v6.21.2
NVIDIA Docs Hub Homepage  NVIDIA Networking  Networking Software  Management Software  NVIDIA UFM Enterprise User Manual v6.21.2  Installing UFM Infra Using Rootless with Podman

On This Page

Installing UFM Infra Using Rootless with Podman

The UFM Infra feature introduces a structured architecture where services are divided into two categories, each deployed differently based on functionality:

  • UFM Infra: A set of persistent infrastructure services that run on all nodes. These services support system-level operations and ensure distributed availability.

  • UFM Enterprise: Services that run exclusively on the master node, responsible for management, orchestration, and user-facing functionality.

For more information on the UFM Infra architecture, refer to UFM Infra.

UFM Infra Rootless Deployment with Podman (Restricted to Oracle Linux Only)

Prerequisites

  1. Download the UFM and plugins bundle tar file to /tmp.

  2. Extract the contents using the command:

    Copy
    Copied!
                
    
            
     tar -xvf <bundle tar>

This archive (tar file) includes the following components:

  • Relevant UFM container image

  • Relevant FAST-API container image

  • Relevant Infra container image (for internal Redis usage). Refer to Redis-Related Configuration for more information.

  • Default plugin bundle for UFM

  • UFM-HA package

HA Installation Requirements

To enable the UFM Infra feature, UFM HA must be installed in a new mode (`external-storage`), using a new product (`enterprise-multinode`).

Additionally, NFS must be configured as follows:

NFS Setup Prerequisites

  • Select a dedicated NFS server to host the shared directories.

  • Create a shared directory on the NFS server for UFM configuration and logs.

  • Install the NFS client on each UFM node if not already present.

Enable HA ports in firewall

If you have firewall rules that blocks non-standard ports, we need to open these ports so high availability services could communicate with each other on the HA nodes. To do so, run these commands:

Copy
Copied!
            

            
firewall-cmd --permanent --add-service=high-availability
# or
firewall-cmd --add-service=high-availability
# and then reload the rules
firewall-cmd --reload


Create and Mount the UFM Directory

Note

At this stage, apply point #2 (Mount the UFM directory) only on the master machine.

Other nodes will be visited for mount later.

  1. Create the UFM directory:

    Copy
    Copied!
                
    
            
     mkdir -p /opt/ufm/files/

  2. Mount the UFM directory:

    • If using NFS 4.2:

      Copy
      Copied!
                  
      
            
       mount -t nfs4 -o context="system_u:object_r:container_file_t:s0" <server>:/shared_folder /opt/ufm/shared_files

    • If using NFS 3:

      Copy
      Copied!
                  
      
            
       mount -t nfs -o vers=3,context="system_u:object_r:container_file_t:s0" <server>:/shared_folder /opt/ufm/shared_files

  3. Ensure the NFS version and mount options are compatible with the NFS server.

  4. Verify that the following HA packages are installed: pcs, pacemaker, and corosync. Install them if they are missing.

  5. Follow the HA installation steps in Run the HA Installation.

Run the HA Installation

Follow the HA installation instructions at UFM High-Availability Installation and Configuration.

When running the HA installation script, use the following command:

Copy
Copied!
            

            
./install.sh -p enterprise-multinode -l /opt/ufm/shared_files

  • The -l flag must always point to the shared directory path: /opt/ufm/shared_files

  • No need to provide the DRBD disk argument to the installation script.

Installation steps

Check Firewall Status

Copy
Copied!
            

            
systemctl status firewalld


Configure Firewall (if active)

Copy
Copied!
            

            
# check if firewalld is running
   systemctl status firewalld
   # Permanently add port 8443 to firewalld
   firewall-cmd --permanent --add-port=8443/tcp
   # reload firewalld config
   firewall-cmd --reload


Create UFM directory

Copy
Copied!
            

            
mkdir -p /opt/ufm


Create UFM Group

Copy
Copied!
            

            
groupadd ufmadm -g 733


Create UFM Iser

Copy
Copied!
            

            
useradd -d /opt/ufm -m -u 733 -g ufmadm ufmadm


Set directory Ownership

Copy
Copied!
            

            
chown -R ufmadm:ufmadm /opt/ufm
chown -R ufmadm:ufmadm /opt/ufm/shared_files

Add Subid/Subgid Configuration

Copy
Copied!
            

            
echo "ufmadm:100000:65536" >> /etc/subuid
echo "ufmadm:100000:65536" >> /etc/subgid

Enalbe Login Linger for UFM User

Copy
Copied!
            

            
loginctl enable-linger ufmadm

Configure Rootless Podam Storage

Copy
Copied!
            

            
sudo -u ufmadm mkdir -p /opt/ufm/.config/containers
cat <<EOF | sudo -u ufmadm tee /opt/ufm/.config/containers/storage.conf > /dev/null
[storage]
driver = "overlay"
runroot = "/run/user/733"
EOF

Create Create Podman UFM Socket

Copy
Copied!
            

            
cat <<EOF > /usr/lib/systemd/system/podman-ufm.socket
[Unit]
Description=Podman API Socket For Nvidia UFM
 
[Socket]
SocketUser=ufmadm
SocketGroup=ufmadm
ListenStream=%t/podman-ufm/podman-ufm.sock
SocketMode=0660
 
[Install]
WantedBy=sockets.target
EOF


Create Podman UFM Service

Copy
Copied!
            

            
cat <<EOF > /usr/lib/systemd/system/podman-ufm.service
[Unit]
Description=Podman API Service for Nvidia UFM
Requires=podman-ufm.socket
After=podman-ufm.socket
StartLimitIntervalSec=0
 
[Service]
Delegate=true
Type=exec
User=ufmadm
Group=ufmadm
KillMode=process
Environment=LOGGING="--log-level=info"
ExecStart=/usr/bin/podman \$LOGGING system service
LimitMEMLOCK=infinity
 
[Install]
WantedBy=default.target
EOF


Create Podman Cleanup Service

Copy
Copied!
            

            
cat <<EOF > /usr/lib/systemd/system/podman-ufm-cleanup.service
[Unit]
Description=podman-ufm-cleanup - clean stuck rootless containers at boot
After=podman-ufm.service
Before=ufm-enterprise.service
 
[Service]
Type=oneshot
User=ufmadm
Group=ufmadm
ExecStart=/usr/bin/podman system migrate
 
[Install]
WantedBy=multi-user.target
EOF


Enable and Start Podman Services

Copy
Copied!
            

            
systemctl daemon-reload
systemctl enable --now podman-ufm.socket
systemctl enable --now podman-ufm.service
systemctl enable --now podman-ufm-cleanup.service


Create Udev Rules for InfiniBand Devices

Copy
Copied!
            

            
cat <<EOF > /etc/udev/rules.d/70-umad.rules
KERNEL=="umad*", SUBSYSTEM=="infiniband_mad", MODE="0600", OWNER="ufmadm", GROUP="ufmadm"
KERNEL=="issm*", SUBSYSTEM=="infiniband_mad", MODE="0600", OWNER="ufmadm", GROUP="ufmadm"
EOF
 
udevadm control --reload-rules
udevadm trigger


Clean and Create UFM Directories

Copy
Copied!
            

            
rm -rf /opt/ufm/systemd
sudo -u ufmadm mkdir -p /opt/ufm/ufm_plugins_data
sudo -u ufmadm mkdir -p /opt/ufm/systemd
sudo -u ufmadm mkdir -p /opt/ufm/etc/apache2


Load UFM Image and Extract Version

Copy
Copied!
            

            
# Extract UFM version from filename (e.g., ufm_6.22.0-7.ubuntu24.x86_64-docker.img.gz -> 6_22_0_7)
UFM_VERSION=$(basename "$UFM_IMAGE_FILE" | sed 's/ufm_\([0-9][^.]*\.[^.]*\.[^.]*-[^.]*\)\.ubuntu.*/\1/' | tr '.-' '_')
echo "UFM Version: $UFM_VERSION"
 
# Load the UFM image
sudo -u ufmadm podman load -i "$UFM_IMAGE_FILE"


Copy
Copied!
            

            
# Create version-specific directory in shared storage
sudo -u ufmadm mkdir -p /opt/ufm/shared_files/ufm-${UFM_VERSION}
 
# Remove existing files link if it exists
rm -f /opt/ufm/files
 
# Create soft link to version-specific directory
sudo -u ufmadm ln -s /opt/ufm/shared_files/ufm-${UFM_VERSION} /opt/ufm/files
 
# Verify the soft link
ls -la /opt/ufm/files


Run UFM Installer

Copy
Copied!
            

            
sudo -u ufmadm podman run -it --rm --name=ufm_installer \
                          -v /run/podman-ufm/podman-ufm.sock:/var/run/docker.sock \
                          -v /opt/ufm/:/installation/ufm_files/ \
                          -v /opt/ufm/files:/installation/ufm_files/files \
                          -v /opt/ufm/systemd:/etc/systemd_files/ \
                          mellanox/ufm-enterprise:latest \
                          --install \
                          --fabric-interface ib0 \
                          --rootless \
                          --plugin-path /opt/ufm/ufm_plugins_data \
                          --ufm-user ufmadm \
                          --ufm-group ufmadm \
                          --ufm-infra
```
**Note**: Replace `ib0` with your actual InfiniBand interface name, if it is not the default ib0.
**Note**: - All other UFM install flags are supported and can be added to the command.


Load Redis Image

Copy
Copied!
            

            
Load the given Redis image (in case you are not using external Redis)
sudo - u ufmadm load -i "<PATH TO GIVEN REDIS IMAGE>"


Load Fast API Plugin Image

Copy
Copied!
            

            
sudo - u ufmadm run --hostname $HOSTNAME --rm --name=ufm_plugin_mgmt --entrypoint="" \
    -v /run/podman-ufm/podman-ufm.sock:/var/run/docker.sock \
    -v /opt/ufm/files:/opt/ufm/shared_config_files \
    -v /dev/log:/dev/log \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /lib/modules:/lib/modules:ro \
    -v /opt/ufm/ufm_plugins_data:/opt/ufm/ufm_plugins_data \
    -e UFM_CONTEXT=ufm-infra \
    mellanox/ufm-enterprise:latest \
    /opt/ufm/scripts/manage_ufm_plugins.sh add -p fast_api -t ${FAST_API_VERSION} -c ufm-infra


Install Service Files

Copy
Copied!
            

            
mv /opt/ufm/systemd/ufm-enterprise.service /etc/systemd/system/ufm-enterprise.service
mv /opt/ufm/systemd/ufm-infra.service /etc/systemd/system/ufm-infra.service
systemctl daemon-reload

To start UFM as a standalone instance, run:

Copy
Copied!
            

            
systemctl daemon-reload 
systemctl start ufm-infra 
systemctl start ufm-enterprise


Running in HA Mode

Note

Do not manually start any services.

  1. Ensure UFM and UFM-HA are installed on all nodes as described in the above sections.

  2. Mount /opt/ufm/files on all standby nodes as described point #2 (Mount the UFM directory)

  3. On one node, edit the HA configuration file:

    Copy
    Copied!
                
    
            
    /etc/ufm_ha/ha_nodes.cfg

    Fill each node parameters

    Copy
    Copied!
                
    
            
    [Node.1]
# valid role options: master/standby
role = master
# Mandatory
primary_ip =
# Mandatory if dual_link = true 
secondary_ip =
 
[Node.2]
role = standby
primary_ip =
secondary_ip =
 
[Node.3]
role = standby
primary_ip =
secondary_ip =

  4. Ensure the file sync mode is set to external-storage, and that the shared file system is mounted prior to HA configuration.

    Copy
    Copied!
                
    
            
    [FileSync]
# valid options are: drbd/external-storage
# in case of external-storage the user MUST mount the files system PRIOR to ha configuration
mode = external-storage

  5. Copy the edited file to all nodes at the same path.

  6. Configure the cluster, starting from standby nodes and ending with the master node:

    Copy
    Copied!
                
    
            
    ufm_ha_cluster config -p <password>

    Note

    Use the same password on all nodes.

  7. After finishing the configuration on all nodes, run:

    Copy
    Copied!
                
    
            
    ufm_ha_cluster status

  8. Start the cluster:

    Copy
    Copied!
                
    
            
    ufm_ha_cluster start

  9. Check cluster status again to ensure all services have started successfully.
© Copyright 2025, NVIDIA. Last updated on Sep 30, 2025.
content here