IMEX Daemon Command Service#
The nvidia-imex daemon can also be configured to have a command service enabled (refer to IMEX Service CMD Utility Service Configuration). Enabling this feature allows you to use the nvidia-imex-ctl tool, which is also installed as part of the IMEX package. This tool facilitates querying the state of one (or all) IMEX instances and providing a convenient way to assess the IMEX cluster’s status without manually inspecting all log files.
The nvidia-imex-ctl Tool#
The nvidia-imex-ctl tool connects to the IMEX Service command service.
Here is an example of the usage:
nvidia-imex-ctl --help
Usage: nvidia-imex-ctl [-c <nvidia-imex config.cfg path>] [-t <timeout_seconds>] [-u <unix domain socket path>] <-n|-N|-q|-D|-ping|--get_error_counters|--subscribe_error_events|-sanity> [-j][-H][-e][-p]
Operations (specify only one; defaults to -N if omitted):
-N Gets the full status of the entire IMEX domain. Requires config file
(default or via -c).
-n Continuously monitor the entire IMEX domain status. Requires config
file (default or via -c).
-q Query the status of the IMEX daemon once and return.
-D Show detailed information of the IMEX nodes.
-ping Request each IMEX node to ping all other nodes in the IMEX domain and return the results
with a default ping timeout of 2 seconds and a maximum timeout of 60 seconds. Default: false.
--get_error_counters Get error counters from all nodes in the IMEX domain.
Shows aggregated error counts by category and type since startup.
--subscribe_error_events Subscribe to real-time error events from all nodes
in the IMEX domain. Streams error events as they occur.
-sanity Run a sanity check on the IMEX domain.
Modifiers:
-c Path to nvidia-imex configuration file. Default: "/etc/nvidia-imex/config.cfg".
-H Show hostname alongside node IP, works with -n, -N, -D. Default: false.
-j Output results of -n/-N/-D as JSON (for streaming operations, prints one JSON object per line). Default: false.
-u Unix domain socket to use for -q command. Default: "".
-t Timeout in seconds for non-subscription operations. Default: operation-specific.
IMEX_TEST_CTRL operations:
-e Get the list of cached exported objects, output as JSON. Default: false.
-p Get the list of pending requests, output as JSON. Default: false.
Note
The available nvidia-imex-ctl options vary by IMEX version and build. Run nvidia-imex-ctl –help to confirm the options available in your installed version.`
Connect to the IMEX Services#
nvidia-imex-ctl can connect to the IMEX CMD service by using TCP/IP or a UNIX domain socket. Use TCP/IP for IMEX domains and a UNIX domain socket for local queries.
Use the following options:
-c <nvidia-imex-config-path>: Reads the IMEX configuration file and connects to the IMEX CMD service for each node in the domain.-u <unix-domain-socket-path>: Connects to a local IMEX CMD service through a UNIX domain socket. Use this option with-q.
Operations#
This section provides information about the operations.
Full Domain Monitoring and Querying#
The full domain monitoring/querying commands (-n/-N) connect nvidia-imex-ctl to each IMEX daemon in the domain as read in from the configuration file specified by the -c command. The process retrieves or subscribes to status updates, including the status of the IMEX daemon status and the interconnectivity state matrix between the iMEX daemons in the domain. In this guide, information about the subscription/monitoring command (-n) is provided, the output for the one-time query command (-N) is the same, except this command will only retrieve the snapshot and not have indicators for state changes.
Here is an example of running a 2-node NO GPU nvidia-imex domain in monitoring mode (-n) with -H enabled to show hostnames:
./nvidia-imex-ctl -H -n -c nodaemon.cfg
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
R - Recovering - Connection was lost, but clean up has not yet been triggered.
D - Disconnected - Connection was lost, and clean up has been triggreed.
A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication.
!V! - Version mismatch, communication disabled.
!M! - Node map mismatch, communication disabled.
!A! - Authentication error, communication disabled.
C - Connected - Ready for operation
In monitoring mode, changes in state will be denoted by * *
3/18/2025 12:41:51.570
Nodes:
Node #0 * 10.127.59.221 * - UNAVAILABLE - Version: - Hostname: imex-mini-1
Node #1 - 10.127.59.222 - UNAVAILABLE - Version: - Hostname: imex-mini-2
Nodes From\To 0 1
0 I I
1 I I
Domain State: DOWN
3/18/2025 12:42:37.613
Nodes:
Node #0 * 10.127.59.221 * - *READY* - Version: NO_GPU - Hostname: imex-mini-1
Node #1 - 10.127.59.222 - UNAVAILABLE - Version: - Hostname: imex-mini-2
Nodes From\To 0 1
0 *C* *N*
1 I I
Domain State: DEGRADED
3/18/2025 12:42:41.938
Nodes:
Node #0 * 10.127.59.221 * - READY - Version: NO_GPU - Hostname: imex-mini-1
Node #1 - 10.127.59.222 - *READY* - Version: NO_GPU - Hostname: imex-mini-2
Nodes From\To 0 1
0 C *C*
1 *C* *C*
Domain State: UP
Note
The asterisk (*) around an IP address in the node list marks the node from which the nvidia-imex-ctl command is being run.
Run domain query mode (-N) to retrieve the current state and exit. The following sample output is from an 8-node cluster using driver version 580.00 with -H; two nodes have missing hostnames.
Failed to look up hostname for Node #4 IP address: 10.76.188.218
Failed to look up hostname for Node #7 IP address: 10.76.191.240
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
R - Recovering - Connection was lost, but clean up has not yet been triggered.
D - Disconnected - Connection was lost, and clean up has been triggreed.
A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication.
!V! - Version mismatch, communication disabled.
!M! - Node map mismatch, communication disabled.
!A! - Authentication error, communication disabled.
C - Connected - Ready for operation
3/18/2025 19:17:50.187
Nodes:
Node #0 * 10.76.179.115 * - READY - Version: 580.00 - Hostname: imex-compute0.nvidia.com
Node #1 - 10.76.183.30 - READY - Version: 580.00 - Hostname: imex-compute1.nvidia.com
Node #2 - 10.76.184.132 - READY - Version: 580.00 - Hostname: imex-compute2.nvidia.com
Node #3 - 10.76.186.255 - READY - Version: 580.00 - Hostname: imex-compute3.nvidia.com
Node #4 - 10.76.188.218 - READY - Version: 580.00 - Hostname: N/A
Node #5 - 10.76.189.233 - READY - Version: 580.00 - Hostname: imex-compute5.nvidia.com
Node #6 - 10.76.190.23 - READY - Version: 580.00 - Hostname: imex-compute6.nvidia.com
Node #7 - 10.76.191.240 - READY - Version: 580.00 - Hostname: N/A
Nodes From\To 0 1 2 3 4 5 6 7
0 C C C C C C C C
1 C C C C C C C C
2 C C C C C C C C
3 C C C C C C C C
4 C C C C C C C C
5 C C C C C C C C
6 C C C C C C C C
7 C C C C C C C C
Domain State: UP
When you include the -j option with -n, -N or -D, the output will be in JSON format.
Query Full Domain Details#
The -D operation is similar to -N but returns more detailed information about the IMEX domain.
The following example shows the output:
./nvidia-imex-ctl -c config.cfg -D
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
R - Recovering - Connection was lost, but clean up has not yet been triggered.
D - Disconnected - Connection was lost, and clean up has been triggreed.
A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication.
!V! - Version mismatch, communication disabled.
!M! - Node map mismatch, communication disabled.
!A! - Authentication error, communication disabled.
!R! - Not yet connected, but blocking the service from proceeding past WAITING_FOR_RECOVERY.
C - Connected - Ready for operation
1/6/2026 10:23:00.866
Nodes:
Node #0 * 127.0.0.1 * - READY - Version: NO_GPU_TESTLIB
Listening Port: 50000
Uptime: 0 days, 00:12:57
Auth Encryption: OFF
GPU Details: Num GPUs: 2
GPU ID: 524544 UUID: GPU-038afcc5-34ac-4a07-ee59-c190c05539ce
IB GUID: 2934173DF8E6BDD60000000000000000
Cluster UUID: 2934173DF8E6BDD62934173DF8E6BDD6
Probe State: Complete
Clique ID: 32766
Fabric Health Mask: A9000000
Fabric Health Summary: Healthy
GPU ID: 590080 UUID: GPU-df161f93-4170-9d8d-b358-549bac9b9311
IB GUID: AE61348D9E4632910000000000000000
Cluster UUID: 00000000000000000000000000000000
Probe State: In Progress
Clique ID: 32766
Fabric Health Mask: A9000000
Fabric Health Summary: Limited Capacity
Connections:
Node 1: 192.168.0.2 - Never Connected - Disconnected for 777s Reconnects: 0
Node 0: 127.0.0.1 - Connected - Connected for 777s Reconnects: 0
Node #1 - 192.168.0.2 - UNAVAILABLE - Version:
- Node details not available.
Nodes From\To 0 1
0 C N
1 I I
Domain State: DEGRADED
Note
IMEX reports the Auth Encryption line and the GPU Details block only when IMEX_CMD_ALLOW_SENSITIVE_INFO=1 on the queried node. This value is enabled by default. When it is set to 0, the output displays - Sensitive details not available due to configuration. The Listening Port and Uptime fields remain available.
The setting comes from each queried node, not the node on which you run nvidia-imex-ctl. In a domain with mixed settings, some nodes might report these details while others do not.
Run a Ping Matrix Test#
The -ping operation sends ping requests between all IMEX nodes and displays a matrix of round-trip times. Use the results to identify unreachable nodes and timed-out requests.
The following example shows the output:
./nvidia-imex-ctl -c config.cfg -ping
Ping Stats:
===========
0 1 2 3
0 - 0.3578ms 0.3016ms T
1 0.2039ms - 0.2473ms T
2 0.2703ms 0.2158ms - T
3 X X X -
Legend:
X = No response from node.
T = Timeout on ping between nvidia-imex nodes.
Value: time (in milliseconds) for ping to complete between nodes.
Error Counters#
The IMEX CMD service includes error counters to track failures in import, unimport, and connection operations across the IMEX domain. Thes counters provide the following information:
Aggregate totals for all responding nodes.
Totals for each node.
Recent error details retained for each node.
Note
Error counters increase throughout runtime and do not reset while the daemon is running. The daemon periodically clears the rolling Recent Errors list.
Get Error Counters#
Use --get_error_counters to query the current error counter snapshot from all nodes listed in IMEX_NODE_CONFIG_FILE and print an aggregated summary.
Here is an example:
nvidia-imex-ctl -c /etc/nvidia-imex/config.cfg --get_error_counters
Querying 2 nodes...
✓ Node 0: Response received
✓ Node 1: Response received
IMEX Cluster-Wide Error Summary
===============================
Cluster Aggregate Totals:
Import Errors:
NOT_FOUND: 0
TIMEOUT: 0
PERMISSION_ERROR: 20
INTERNAL_ERROR: 0
Total: 20
Unimport Errors:
NOT_FOUND: 20
TIMEOUT: 0
PERMISSION_ERROR: 0
INTERNAL_ERROR: 0
Total: 20
Connection Errors:
AUTHENTICATION_ERROR: 0
VERSION_MISMATCH: 0
NODES_MISMATCH: 0
CONNECTION_LOST: 0
Total: 0
Cluster Total: 40 errors
Per-Node Summary:
Node 0: 20 errors (Import: 10, Unimport: 10, Connection: 0)
Node 1: 20 errors (Import: 10, Unimport: 10, Connection: 0)
Recent Errors on Node 0: 2
[2025-12-13 06:05:48.206] To Node 1: IMPORT_ERROR.IET_PERMISSION_ERROR
Description: "Import failed: PERMISSION_DENIED"
Driver Error: "Requested DMA is in use [NV_ERR_DMA_IN_USE]"
[2025-12-13 06:05:48.209] To Node 1: UNIMPORT_ERROR.IET_NOT_FOUND
Description: "Unimport failed: UNIMPORT_OBJECT_NOT_FOUND"
Command completed successfully.
The error types include:
import and unimport errors use the listed
ImportErrorTypevalues; aggregate and streamed output omitIET_;Recent Errorsdisplays the raw names.Connection error types:
AUTHENTICATION_ERROR,VERSION_MISMATCH,NODES_MISMATCH,CONNECTION_LOST
Subscribe to Error Events#
Use --subscribe_error_events to stream real-time error events from all nodes in the domain. This command runs until it is interrupted.
Here is an example:
nvidia-imex-ctl -c /etc/nvidia-imex/config.cfg --subscribe_error_events
Establishing error event streams to 2 nodes...
Streaming Error Events from cluster...
[2025-12-13 06:05:48.205] [From Node 1] IMPORT_ERROR.PERMISSION_ERROR to Node 1
Description: "Import failed: PERMISSION_DENIED"
Driver Error: "Requested DMA is in use [NV_ERR_DMA_IN_USE]"
[2025-12-13 06:05:48.206] [From Node 1] UNIMPORT_ERROR.NOT_FOUND to Node 1
Description: "Unimport failed: UNIMPORT_OBJECT_NOT_FOUND"
Run a Domain Sanity Check#
The -sanity operation queries every node listed in IMEX_NODE_CONFIG_FILE for its detailed status (the same information that -D prints). It then compares the reported settings across the domain and prints a validation report. Use it to catch misconfigurations that keep a domain from forming correctly, such as one node listening on a different port or a node using a different encryption mode than the rest.
The report contains five checks:
Server port: every node must report the same
IMEX_SERVER_PORT.Authentication and encryption: every node must report the same
IMEX_ENABLE_AUTH_ENCRYPTIONsetting.Encryption mode: every node must report the same
IMEX_AUTH_ENCRYPTION_MODE.Cluster UUID: every GPU in the domain must report the same cluster UUID.
Fabric Partition ID: every GPU in the domain must report the same fabric partition ID.
Each check reports one of the following results:
PASS: all nodes (or all GPUs) reported the same value, and every node in the domain contributed data to the check.
PARTIAL PASS: all of the nodes that reported data agreed, but at least one node was skipped, so the check is inconclusive for the domain as a whole.
AWARNINGblock lists how many nodes were checked and which node IDs were skipped.
A node is skipped when it did not return node details (typicallyUNAVAILABLEor slower than the timeout) or when it did not return sensitive information.FAIL: more than one distinct value was found. The mismatching values are listed along with the node IDs (and, for the GPU checks, the individual GPUs) that reported each value, so you can tell which node is the outlier.
Every check is followed by a coverage line: All <N> nodes checked. when nothing was skipped, or the WARNING block described above.
Here is an example of a healthy 2-node domain, with the per-node detail output abbreviated:
./nvidia-imex-ctl -c config.cfg -sanity
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
...
C - Connected - Ready for operation
1/6/2026 10:23:00.866
Nodes:
Node #0 * 10.76.179.115 * - READY - Version: 580.00 - IMEX Version: 580.00
Listening Port: 50000
Uptime: 0 days, 01:12:57
Auth Encryption: OFF
GPU Details: Num GPUs: 1
GPU ID: 524544 UUID: GPU-038afcc5-34ac-4a07-ee59-c190c05539ce
IB GUID: 2934173DF8E6BDD60000000000000000
Cluster UUID: 2934173DF8E6BDD62934173DF8E6BDD6
Probe State: Complete
Fabric Partition ID: 0
Clique ID: 1
Fabric Health Summary: Healthy
Connections:
Node 0: 10.76.179.115 - Connected - Connected for 4377s Reconnects: 0
Node 1: 10.76.183.30 - Connected - Connected for 4377s Reconnects: 0
...
Nodes From\To 0 1
0 C C
1 C C
Domain State: UP
= Server ports consistency across the IMEX domain:
PASS
All 2 nodes checked.
= Checking enable auth encryption (IMEX_ENABLE_AUTH_ENCRYPTION) consistency across the IMEX domain:
PASS
All 2 nodes checked.
= Checking encryption mode (IMEX_AUTH_ENCRYPTION_MODE) consistency across the IMEX domain:
PASS
All 2 nodes checked.
= Cluster UUID consistency across all GPUs in IMEX domain:
PASS
All 2 nodes checked.
= Fabric Partition ID consistency across all GPUs in IMEX domain:
PASS
All 2 nodes checked.
The next example shows the validation report from a 3-node domain with problems: node 1 is listening on a different port, node 1 is configured with a different encryption mode, and node 2 did not return sensitive information, which makes the remaining checks inconclusive.
= Server ports consistency across the IMEX domain:
FAIL: Server port mismatch found:
Server port: 50000, Node IDs: 0 (imex-compute0.nvidia.com), 2 (imex-compute2.nvidia.com)
Server port: 50001, Node IDs: 1 (imex-compute1.nvidia.com)
All 3 nodes checked.
= Checking enable auth encryption (IMEX_ENABLE_AUTH_ENCRYPTION) consistency across the IMEX domain:
PARTIAL PASS
WARNING: 2 of 3 nodes checked.
The following nodes do not have node_info_sensitive populated (Please ensure IMEX_CMD_ALLOW_SENSITIVE_INFO is set to 1 in the IMEX config file to populate node_info_sensitive):
2 (imex-compute2.nvidia.com)
= Checking encryption mode (IMEX_AUTH_ENCRYPTION_MODE) consistency across the IMEX domain:
FAIL: Encryption mode mismatch found:
Encryption mode: GSS_AUTH_ENCRYPT, Node IDs: 1 (imex-compute1.nvidia.com)
Encryption mode: SSL_TLS, Node IDs: 0 (imex-compute0.nvidia.com)
WARNING: 2 of 3 nodes checked.
The following nodes do not have node_info_sensitive populated (Please ensure IMEX_CMD_ALLOW_SENSITIVE_INFO is set to 1 in the IMEX config file to populate node_info_sensitive):
2 (imex-compute2.nvidia.com)
= Cluster UUID consistency across all GPUs in IMEX domain:
FAIL: Cluster UUID mismatch found:
Cluster UUID: 2934173DF8E6BDD62934173DF8E6BDD6, GPUs:
{ Node ID: 0 (imex-compute0.nvidia.com), GPU ID: 524544, GPU UUID: GPU-038afcc5-34ac-4a07-ee59-c190c05539ce }
Cluster UUID: 00000000000000000000000000000000, GPUs:
{ Node ID: 1 (imex-compute1.nvidia.com), GPU ID: 590080, GPU UUID: GPU-df161f93-4170-9d8d-b358-549bac9b9311 }
WARNING: 2 of 3 nodes checked.
The following nodes do not have node_info_sensitive populated (Please ensure IMEX_CMD_ALLOW_SENSITIVE_INFO is set to 1 in the IMEX config file to populate node_info_sensitive):
2 (imex-compute2.nvidia.com)
Caution
Only the server port check works on data that every node always reports. The other four checks (auth encryption, encryption mode, cluster UUID, and fabric partition ID) read the sensitive node information that a node only returns when its own config file sets IMEX_CMD_ALLOW_SENSITIVE_INFO=1 (the default). A node that has it set to 0 is counted as skipped for those four checks, which turns an otherwise clean result into PARTIAL PASS and can hide a real mismatch on that node, so set it to 1 on every node before you rely on the sanity report.
Warning
-sanity exits with status 0 even when a check reports FAIL, so scripts must inspect the output rather than rely on the exit code.
Note
The -t option bounds how long the tool waits for node responses (1 second by default); nodes that do not answer in time are reported as skipped instead of validated, -H adds hostnames to the node list, and -j renders only the domain status portion as JSON, while the validation report is always plain text.