Signal handling#

Shutting down#

The following signals will all cause IMEX to start shutting down:

  • SIGQUIT

  • SIGINT

  • SIGTERM

When shutting down, the following steps will happen:

  1. The signaled nvidia-imex instance will notify other nodes of the shutdown.

  2. Existing exports from the signaled node will be terminated on the corresponding remote importing node.

Dump in-memory verbose logging to file#

The nvidia-imex service will buffer any messages that are above the current logging setting to an in-memory buffer that can be dumped to file when signalled via SIGHUP. This buffer is also automatically flushed when IMEX shuts down. As an example, if logging is currently set to 3 (Warning), INFO messages will not be recorded to the regular log file, but they will be captured in the verbose buffer to be available if flushed.

Reloading SSL credentials and restarting connections#

SIGUSR1 can be used to trigger the reloading of SSL credentials, and also restart all outgoing client connections. The logic is broken up based on if SSL/mTLS is enabled.

Note: SIGUSR1 will only be processed once every 5 seconds. Any repeated SIGUSR1 received within the 5 second window will be ignored.

SSL/mTLS enabled#

If SSL/mTLS is enabled and configured to be read from a file, then IMEX can reload the credentials when triggered via SIGUSR1, with the following steps:

  1. Attempt to reload the files configured by the IMEX_SERVER_KEY, IMEX_SERVER_CERT, IMEX_SERVER_CERT_AUTH, IMEX_CLIENT_KEY, IMEX_CLIENT_CERT, IMEX_CLIENT_CERT_AUTH parameters.

  2. Notify all other nvidia-imex daemons that they need to reload their credentials

  3. Shut down existing outgoing client connections and re-start them with the new reloaded credentials.

Note: If step 1 fails, step 2 and 3 will be skipped.

Example of a configuration that supports SSL/mTLS credential reload: (See IMEX Service Secure Connections Configuration for configuration details.)

IMEX_ENABLE_AUTH_ENCRYPTION=1
IMEX_AUTH_ENCRYPTION_MODE=SSL_TLS
IMEX_AUTH_SOURCE=FILE
IMEX_SERVER_KEY=/etc/nvidia-imex/certs/server.key
IMEX_SERVER_CERT=/etc/nvidia-imex/certs/server.crt
IMEX_SERVER_CERT_AUTH=/etc/nvidia-imex/certs/root.crt
IMEX_CLIENT_KEY=/etc/nvidia-imex/certs/client1.key
IMEX_CLIENT_CERT=/etc/nvidia-imex/certs/client1.crt
IMEX_CLIENT_CERT_AUTH=/etc/nvidia-imex/certs/root.crt

The recommendation for using this signal to reload credentials would be to:

  1. Update the credential files across the entire IMEX domain

  2. Signal one of the nvidia-imex daemons with SIGUSR1 to trigger the entire IMEX domain to reload their credentials.

Non-SSL/mTLS handling#

If we are not using SSL/mTLS, then SIGUSR1 will just shut down all outgoing connections and re-establishing them. This can be useful if a connection has been blocked due to an authentication or authorization error and the IMEX_UNAUTHORIZED_RECONNECT_INTERVAL_SEC has been set to never automatically reconnect. Then SIGUSR1 can be used to restart and re-attempt the connections and re-authorization.