Getting Started with IMEX#
Basic Components#
This section provides information about the basic components in the IMEX service.
The IMEX Service#
The core component of IMEX is implemented as a stand-alone executable that runs as a UNIX daemon process. The IMEX installation package installs the required core components and registers the service as a systemd service called nvidia-imex.
Note
Starting with CUDA 13.5 and driver version 620.*, nvidia-imex uses its own versioning, starting with 2.0.0.*, with the driver install containing a libnvidia-imex.so.* shared library.
Prior releases still include the nvidia-imex version that matches the installed driver release.
The IMEX Control App#
In addition to the IMEX service, the installation package also installs the nvidia-imex-ctl. You can use this tool to query the state of the IMEX service (refer to the nvidia-imex-ctl tool for more information).
Supported Platforms#
IMEX currently supports the following products and environments:
Hardware architectures:
x86_64
aarch64
Here are the NVLink network server architectures:
NVIDIA GB200 NVL72
OS Environment
IMEX is supported on the following major Linux OS distributions:
RHEL/CentOS 7.x and RHEL/CentOS 8.x
Ubuntu 20.04.x, Ubuntu 22.04.x and Ubuntu 24.0x
Other NVIDIA Software Packages#
To run the IMEX service, the target system must include a compatible Data Center GPU Driver package for GB200 systems.
Note
During initialization, the IMEX service verifies that the currently loaded kernel driver stack version is compatible. If the loaded version is not compatible, the service aborts the startup process.
Installation#
This section provides information about installing the IMEX service.
On DGX NVLink Multi-Node Systems#
On DGX NVLink Multi-Node systems, the IMEX service is preinstalled as part of the supported DGX OS package.
On NVIDIA HGX NVLink Multi-Node Systems#
On NVIDIA HGX NVLink Multi-Node systems, the IMEX service needs to be installed on the compute node or on the guest VMs that were created on the compute nodes in a virtualized environment.
The IMEX package is available through the CUDA network repository. Refer to the NVIDIA Driver Installation Quickstart Guide for more information about setting up your system’s package manager and downloading packages from a CUDA network repository.
After the package manager network repositories are set up, use the following distro-specific IMEX installation commands:
Note
Substitute the <driver-branch> commands with the required NVIDIA driver branch number for qualified datacenter drivers (for example, 550).
For Debian and Ubuntu based OS distributions:
sudo apt-get install nvidia-imex-<driver-branch>
For Red Hat Enterprise Linux 8 based OS distributions:
sudo dnf module install nvidia-imex:<driver-branch>/fm
SUSE Linux based OS distributions:
sudo zypper install nvidia-imex-<driver-branch>
Note
As part of the installation, the IMEX service unit file (nvidia-imex.service) is copied to systemd. However, the system administrator must manually enable and start the IMEX service. |
Managing the IMEX Service#
This section provides information about managing the IMEX service.
Basic Configuration#
The nvidia-imex service installer creates the default configuration file at /etc/nvidia-imex/config.cfg. It does not create the domain configuration file at /etc/nvidia-imex/nodes_config.cfg.
Add the IP address or hostname of each node in the IMEX domain to nodes_config.cfg.
For additional information about configuration, see the configuration options page
Start the IMEX#
To start IMEX, run the following command.
# For Linux based OS distributions
sudo systemctl start nvidia-imex
Stopping the IMEX Service#
To stop IMEX, run the following command.
# For Linux based OS distributions
sudo systemctl stop nvidia-imex
Checking the IMEX Service Status#
To check the IMEX service status, run the following command.
# For Linux based OS distributions
sudo systemctl status nvidia-imex
For a detailed IMEX service status, use the nvidia-imex-ctl tool (refer to The nvidia-imex-ctl for more information).
Enabling the IMEX Service to Automatically Start at Boot#
To enable the IMEX service to automatically start at boot, run the following command.
# For Linux based OS distributions
sudo systemctl enable nvidia-imex
Disabling the IMEX Service Automatically Start at Boot#
To disable the IMEX service to automatically start at boot, run the following command.
# For Linux based OS distributions
sudo systemctl disable nvidia-imex
Checking IMEX Service System Log Messages#
To check the IMEX system log messages, run the following command.
# For Linux based OS distributions
sudo journalctl -u nvidia-imex
IMEX Service Startup Options#
IMEX supports the following CLI options:
. /nvidia-imex -h
/usr/bin/nvidia-imex –h
[-h | --help]: Displays help information.
[-v | --version]: Displays the imex version and exit.
[-c | --config]: Provides imex config file path/name which controls all the config options.
[-nogpu | --nogpu]: Run in "NO GPU" mode, allowing testing of nvidia-imex communication and configuration without requiring or using a GPU.
[-p | --port]: Specifies the port for IMEX peer communication. If specified, this overrides the IMEX\_SERVER\_PORT in config file.
Most of the IMEX configurable parameters and options are specified through a text config file. When you install IMEX, a default config file will be copied to a predefined location, and the file will be used by default. To use a different config file location, specify the same parameters and options using the [-c | –config] command-line argument.
Note
On Linux-based installations, the default IMEX config file is located in the /etc/nvidia-imex/ directory. If you modify the default config file, subsequent IMEX package updates will offer options to manage it, including Merge, Keep, and Overwrite.
IMEX Service File#
This section provides information about the IMEX service file.
On Linux-Based Systems#
On Linux-based systems, the installation package will register the IMEX service using the following systemd service unit file. To change the IMEX service start-up options, modify this service unit file in the /lib/systemd/system/nvidia-imex.service directory.
[Unit]
Description=NVIDIA IMEX service
After=network-online.target
Requires=network-online.target
[Service]
User=root
PrivateTmp=false
Type=forking
TimeoutStartSec=infinity
ExecStart=/usr/bin/nvidia-imex -c /etc/nvidia-imex/config.cfg
LimitCORE=infinity
[Install]
WantedBy=multi-user.target
Running IMEX Service as Non-Root#
On Linux-based systems, for memory fabric events the IMEX service requires administrative (root) privileges to register with the GPU driver. However, system administrators and advanced users can complete the following steps to run IMEX from a non-root account:
If the IMEX service is running, stop it.
IMEX requires access to the following directory/file:
/etc/nvidia-imex/: This directory accesses the IMEX configuration files.
/var/log/: This directory saves the IMEX log file.
Update the corresponding directory/file access to the appropriate user or user group.
Change the following proc entry read/write access to point to the user or user group.
/proc/driver/nvidia/capabilities/fabric-imex-mgmt
The fabric-imex-mgmt file contains following entries:DeviceFileMinor: 4323
DeviceFileMode: 256
DeviceFileModify: 1
Note down the value of DeviceFileMinor, for example 4323.
Run the following command.
echo DeviceFileModify: 0 > /proc/driver/nvidia/capabilities/fabric-imex-mgmtChange the read/write access to the appropriate user/group for the following file
/dev/nvidia-caps/nvidia-cap<DeviceFileMinor>Modify the access with the DeviceFileMinor value from step 4.
/dev/nvidia-caps/nvidia-cap4323After the required permissions are assigned, manually start the IMEX service process from the user/user group account.
The NVIDIA driver will create/recreate the above /proc entry when the driver loads, so Repeat steps 1-8 on every driver reload or system boot.
When IMEX is configured as a systemd service, the system administrator must edit the IMEX service unit file to instruct systemd to run the IMEX service from a user/group. This user/group can be specified by using the User= and Group= directives in the [Service] section of the IMEX service unit file. The system administrator must ensure that the proc entry and associated file node permission are changed to the user/user group before the IMEX service starts at system boot time.
Note System administrators can set up necessary udev rules to automate the process of changing these proc entry permissions. |
Running IMEX in NO GPU mode#
You can run IMEX in NO GPU mode with the “–nogpu” command line flag to test nvidia-imex communication and configuration without requiring or using a GPU.
Example:
./nvidia-imex -c config.cfg –nogpu