IMEX Service Config Options#
The configurable parameters and options used by the IMEX Service are specified through a text config file. The following section lists the currently supported configurable parameters and options.
Note
The IMEX configuration file is read during the IMEX service startup. To apply any changes made to the config file, you must restart the IMEX service.
Miscellaneous Config Items#
This section provides information about miscellaneous config items.
Prevent IMEX from Daemonizing#
Config Item
DAEMONIZE=<value>Supported/Possible Values
0: Do not daemonize and run IMEX as a normal process.
1: Run the IMEX process as a Unix daemon.
Default Value
DAEMONIZE=1
IMEX Service Communication Socket Interface#
Config Item
BIND_INTERFACE_IP=<value>Supported/Possible Values
Network interface to listen for the IMEX internal communication/IPC, and this value should be a valid IPv4 or IPv6 address. If this value is empty, it will automatically try to find an IP address in the IMEX_NODE_CONFIG_FILE that matches the IP addresses available via the system interface list.Default Value
BIND_INTERFACE_IP=
IMEX Service Communication TCP Listening Port#
Config Item
SERVER_PORT=<value>Supported/Possible Values
TCP port number for the IMEX internal communication/IPC, and this value should be between 0 and 65535.Default Value
SERVER_PORT=50000
Note
This configuration may be overridden by command line option [-p | –port]. See 2.5 for command line help. |
Note
Occasionally, the IMEX service may fail to start because a required port is already in use. To prevent this issue, set SERVER_PORT outside the ephemeral port range. Alternatively, reserve the ports assigned to SERVER_PORT and IMEX_CMD_PORT.
IMEX Service Node Configuration File Location#
Config Item
IMEX_NODE_CONFIG_FILE=<value>Supported/Possible Values
Name of the file that contains the IP addresses of the nodes that are in this IMEX domain.Default Value
IMEX_NODE_CONFIG_FILE=/etc/nvidia-imex/nodes_config.cfg
Note
This configuration file is not created during the installation. The Administrator must manage and create the file for the nvidia-imex daemon.
IMEX Service Network Interface#
Config Item
NETWORK_INTERFACE=<value>Supported/Possible Values
Name of the network interface used for communication.
If empty, NETWORK_INTERFACE will be determined by matching bind IP to the list of system network interfaces. This configuration is only necessary if the bind IP is present on multiple network interfaces.Default Value
NETWORK_INTERFACE=
IMEX Service Outgoing Network Interface#
Config Item
OUTGOING_NETWORK_INTERFACE=<value>Supported/Possible Values
Name of the network interface used for communication.
If empty, outgoing network interface will be determined automatically.
This is only necessary if you want to force all outgoing connections to use a particular interface.Default Value
OUTGOING_NETWORK_INTERFACE=
IMEX Service Connection Wait Timeout#
Config Item
IMEX_CONN_WAIT_TIMEOUT=<value>Supported/Possible Values
Timeout (in seconds) for IMEX to wait for establishing connection with all other nodes in the IMEX_NODE_CONFIG_FILE. A value of -1 will wait forever.Default Value
IMEX_CONN_WAIT_TIMEOUT=-1
IMEX Node Disconnection Grace Period#
Config Item
IMEX_NODE_DISCONNECTED_GRACE_TIME=<value>Supported/Possible Values
Determines how long to wait after detecting that the IMEX daemon has lost connection to another node before cleaning up imports and exports from that node. If a connection is reestablished before the grace period expires, and IMEX identifies it as the same instance previously connected, no cleanup is required. If IMEX detects a new instance (for example, after the IMEX daemon is restarted), cleanup is triggered immediately, regardless of the grace period.-1: Wait indefinitely.
0: Immediately trigger cleanup.
>0: Number of seconds to wait before triggering cleanup.
Default Value
IMEX_NODE_DISCONNECTED_GRACE_TIME=-1
IMEX Quorum establishment#
Config Item
IMEX_WAIT_FOR_QUORUM=<value>Supported/Possible Values
NONE: Do not wait for any quorum with other nodes.
RECOVERY: If an unsafe nvidia-imex termination occurs, wait until the nodes that had imported from this node to connect, which allows them to safely clean up any potentially hanging references.
Default Value
IMEX_WAIT_FOR_QUORUM=RECOVERY
DSCP override for listening socket and outgoing client connections#
Config Item
IMEX_GRPC_DSCP_OVERRIDE=<value>Supported/Possible Values
0-63: Custom DSCP setting
Default Value
IMEX_GRPC_DSCP_OVERRIDE=0
IMEX Service Secure Connections Configuration#
This section provides information about configuring secure connections in the IMEX service.
Enable/Disable Secure Connections#
Config Item
IMEX_ENABLE_AUTH_ENCRYPTION=<value>Supported/Possible Values
0: Disable encryption and authentication
1: Enable encryption and authentication
Default Value
IMEX_ENABLE_AUTH_ENCRYPTION=0
Note
If this parameter is disabled, the remaining authentication and encryption parameters will not be used at run time.
Control Authentication and Encryption Method#
Config Item
IMEX_AUTH_ENCRYPTION_MODE=<value>Supported/Possible Values
SSL_TLS – Use SSL/TLS: SSL/TLS-specific parameters need to be configured.
GSS_AUTH_ENCRYPT: Use GSSAPI for authentication, integrity, and encryption.
GSS-specific parameters need to be configured.GSS_AUTH_ONLY: Use GSSAPI for authentication and integrity only, and encryption will be disabled.
GSS-specific parameters need to be configured.
Default Value
IMEX_AUTH_ENCRYPTION_MODE=SSL_TLS
Control SSL/TLS Authentication Source#
Config Item
IMEX_AUTH_SOURCE=<value>Supported Values:
This configuration parameter will determine how IMEX retrieves the keys, certificates, and certificate authority for authentication and encryption. If IMEX_ENABLE_AUTH_ENCRYPTION is enabled first, IMEX_AUTH_SOURCE must be configured with one of the following supported values. An empty or unexpected value will prevent initialization.FILE: The provided values are paths to files on the file system.
ENV_PATH: The provided values are environment variables names to retrieve, and the values in the environment variables are treated as paths to files on the file system.
ENV_VAL: The provided values are environment variable names to retrieve, and the values in the environment variables are treated as the actual values for the key/certificate/certificate authority.
Default Value
IMEX_AUTH_SOURCE=
SSL/TLS Security Target Override#
Config Item
IMEX_SECURITY_TARGET_OVERRIDE=<value>Supported/Possible Values
Overrides the target hostname for authentication of the certificates and keys. This allows certificates with common names that do not match the IP addresses provided for the nodes.
Here is an example when the certificate has the subject:"/C=US/ST=CA/L=Santa Clara/O=NVIDIA/OU=Test/CN=localhost"
The certificate validation will expect the connection hostname to be localhost. To override the connection hostname for security purposes to be localhost and allow the connection to succeed, setIMEX_SECURITY_TARGET_OVERRIDE=localhost.Default Value
IMEX_SECURITY_TARGET_OVERRIDE=
GSS Service Name#
Config Item
IMEX_GSS_SERVICE_NAME=<value>Supported/Possible Values
Service Principal name to use.Default Value
IMEX_CMD_ENABLED=host
GSS Timeout#
Config Item
IMEX_GSS_TIMEOUT_SEC=<value>Supported/Possible Values
Specifies the timeout for GSSAPI calls in seconds.-1: Retry indefinitely.
>=0: Number of seconds to wait before triggering clean up.
Default Value
IMEX_GSS_TIMEOUT_SEC=-1
GSS Retry Interval#
Config Item
IMEX_GSS_RETRY_INTERVAL_SEC=<value>Supported/Possible Values
Specifies the interval at which to retry failing GSSAPI calls.>=0: Number of seconds to wait between retries.
Default Value
IMEX_GSS_RETRY_INTERVAL_SEC=5
GSS Security Context Lifetime#
Config Item
IMEX_GSS_SEC_CONTEXT_LIFETIME_SEC=<value>Supported/Possible Values
Specifies the lifetime of the GSSAPI security context.-1: Indefinite, nvidia-imex will not manually expire the security context
>=0: Lifetime (in seconds) before nvidia-imex manually expires the security context.
If the system configuration set an expiration value for security contexts, the value will still take effect, and nvidia-imex will refresh the security context at some point before the expiration occurs.
Default Value
IMEX_GSS_SEC_CONTEXT_LIFETIME_SEC=-1
GSS Failure Handling#
Config Item
IMEX_GSS_SHUTDOWN_ON_FAILURE=<value>Supported/Possible Values
Specifies whether nvidia-imex should shut down when it encounters a fatal error or timeout from GSSAPI1: Shutdown nvidia-imex when a fatal error or timeout occurs.
0: Terminate connection (permanently) to the failing node.
Default Value
IMEX_GSS_SHUTDOWN_ON_FAILURE=1
IMEX Service CMD Utility Service Configuration#
This section provides information about configuring the CMD utility service.
Enable/Disable CMD Utility Service#
Config Item
IMEX_CMD_ENABLED=<value>Supported/Possible Values
0: Do not enable the CMD service.
1: Enable the CMD service, and the service must be used with IMEX_CMD_PORT (and optionally IMEX_CMD_BIND_INTERFACE_IP) and/or IMEX_CMD_UNIX_DOMAIN_PATH.
Default Value
IMEX_CMD_ENABLED=1
CMD Service Bind Port#
Config Item
IMEX_CMD_PORT=<value>Supported/Possible Values
A TCP port to bind the CMD service to. Optionally, the port can be combined with IMEX_CMD_BIND_INTERFACE_IP to only bind to an ip:port. The value is
0-65535.Default Value
IMEX_CMD_PORT=50005
CMD Service Bind IP Address#
Config Item
IMEX_CMD_BIND_INTERFACE_IP=<value>Supported/Possible Values
A TCP IP address to which the CMD service will be bound, and the address must be used with IMEX_CMD_PORT.Default Value
IMEX_CMD_BIND_INTERFACE_IP=
CMD Service UNIX Domain Path#
Config Item
IMEX_CMD_UNIX_DOMAIN_PATH=<value>Supported/Possible Values
A UNIX domain socket path for the command/control service.Default Value
IMEX_CMD_UNIX_DOMAIN_PATH=
CMD Service Sensitive Information Reporting#
Config Item
IMEX_CMD_ALLOW_SENSITIVE_INFO=<value>
IMEX_CMD_ALLOW_SENSITIVE_INFO determines whether the CMD service returns sensitive node information in status responses to nvidia-imex-ctl.
Supported values
0: Withholds sensitive node information. Only non-sensitive node status is returned.
1: Includes sensitive node information in the returned node status.
Sensitive node information includes:
Effective node security settings:
IMEX_ENABLE_AUTH_ENCRYPTIONandIMEX_AUTH_ENCRYPTION_MODEPer-GPU inventory: GPU ID, GPU UUID, IB GUID, cluster UUID, probe state, fabric partition ID, clique ID, and fabric health summary
Non-sensitive node status is always returned. It includes the daemon state, version, listening port, uptime, and connectivity matrix.
The full domain detail query (-D) and domain sanity check (-sanity) operations use the sensitive information. When the value is 0, -D reports the following message for the node:
Sensitive details not available due to configuration.
The -sanity operation cannot perform the encryption and GPU consistency checks for that node. It excludes the node from those checks and changes an otherwise clean result to PARTIAL PASS.
Each queried node evaluates this setting and controls the information that it exposes. If nodes in a domain use different values, the -D and -sanity results are complete for some nodes and incomplete for others.
Default Value
IMEX_CMD_ALLOW_SENSITIVE_INFO=1