IMEX Performance Statistics

The IMEX service is crucial for NVLink Multi-Node CUDA jobs, so we recommend that you monitor IMEX queues and message exchange performance statistics. IMEX has a lightweight, thread-safe framework to periodically log these statistics, which are stored by default in the /var/log/nvidia-imex-stats.log file. This logging mechanism ensures visibility into the IMEX performance, which allows the proactive management and optimization of CUDA jobs.

Statistics logging can be redirected to another location with the STATS_FILE_NAME config option.

• Here are the queue statistics:

  • queueDepth: The number of elements in a queue.

  • oldestDurationMillis: The age of the oldest element in a queue (in msec).

• Here are the message statistics (per-direction, per request-type):

  • avgQueueTimeMillis: The average time spent queued for processing (in msec).

  • avgOperTimeMillis: The average time spent executing the task (in msec).

  • avgTotalTimeMillis: The average total time between queueing to completion (in msec).

Here is a snippet from nvidia-imex-stats.log after a sample application is run:

Performance statistics for last 30 seconds  : {
 "QueueStats": {
  "ProcessorQueue": {
   "queueDepth": "0",
   "oldestDurationMillis": 0
  },
  "PendingFreeQueue": {
   "queueDepth": "0",
   "oldestDurationMillis": 0
  }
 },
 "IncomingMessages": {
  "UnimportResponse": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.01,
   "avgOperTimeMillis": 0.06,
   "avgTotalTimeMillis": 0.07
  },
  "UnicastImportRequest": {
   "msgCount": "7",
   "avgQueueTimeMillis": 1.14,
   "avgOperTimeMillis": 0.22,
   "avgTotalTimeMillis": 1.35
  },
  "Heartbeat": {
   "msgCount": "80",
   "avgQueueTimeMillis": 0.01,
   "avgOperTimeMillis": 0.01,
   "avgTotalTimeMillis": 0.01
  },
  "ImportResponse": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.01,
   "avgOperTimeMillis": 0.02,
   "avgTotalTimeMillis": 0.03
  },
  "UnimportRequest": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.04,
   "avgOperTimeMillis": 0.02,
   "avgTotalTimeMillis": 0.05
  }
 },
 "OutgoingMessages": {
  "UnicastImportRequest": {
   "msgCount": "7",
   "avgQueueTimeMillis": 1.21,
   "avgOperTimeMillis": 0,
   "avgTotalTimeMillis": 1.21
  },
  "UnimportRequest": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.14,
   "avgOperTimeMillis": 0,
   "avgTotalTimeMillis": 0.14
  },
  "ImportResponse": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.19,
   "avgOperTimeMillis": 0,
   "avgTotalTimeMillis": 0.19
  },
  "UnimportResponse": {
   "msgCount": "7",
   "avgQueueTimeMillis": 0.1,
   "avgOperTimeMillis": 0,
   "avgTotalTimeMillis": 0.1
  }
 }
}

Note

For outgoing messages, use avgTotalTimeMillis, which reflects the time to deliver the message.

Note

NVLink P2P memory import and unimport requests expect a response. |

Error Counters and Recent Error Details

In addition to queue depths and message timing, IMEX reports error counters and a rolling window of recent error details in the same 30-second statistics record written to /var/log/nvidia-imex-stats.log.

For on-demand access from the command line, refer to the nvidia-imex-ctl tool and the --get_error_counters / --subscribe_error_events operations.

Error counters (error_counters)

error_counters is a cumulative counter that tracks failures in:

• Import operations

• Unimport operations

• Connection establishment and maintenance

The error counter structure contains:

• error_counters.total: Total errors across all destination peers since the local IMEX daemon started.

• error_counters.per_node: Error totals for each destination node ID. This map is created when needed and includes only nodes for which errors were observed.

The counter maps use integer keys that correspond to enumeration values. When serialized as JSON, the keys appear as strings.

Import and unimport error keys (ImportErrorType)

• 1: IET_NOT_FOUND

• 2: IET_TIMEOUT

• 3: IET_PERMISSION_ERROR

• 4: IET_INTERNAL_ERROR

Connection error keys (ConnectionErrorType)

• 1: CET_AUTHENTICATION_ERROR

• 2: CET_VERSION_MISMATCH

• 3: CET_NODES_MISMATCH

• 4: CET_CONNECTION_LOST

Recent Error Details (error_details_last_window)

error_details_last_window contains error records captured during the most recent statistics interval, which is 30 seconds by default. After the IMEX daemon emits each statistics record, it clears the list for the next interval.

The following example shows the error-related fields in the statistics log:

"error_counters": {
 "total": {
  "import_errors": {
   "1": "0",
   "2": "0",
   "3": "20",
   "4": "0"
  },
  "unimport_errors": {
   "1": "20",
   "2": "0",
   "3": "0",
   "4": "0"
  },
  "connection_errors": {
   "1": "0",
   "2": "0",
   "3": "0",
   "4": "0"
  },
  "total_import_errors": "20",
  "total_unimport_errors": "20",
  "total_connection_errors": "0"
 },
 "per_node": {
  "1": {
   "import_errors": {
    "1": "0",
    "2": "0",
    "3": "20",
    "4": "0"
   },
   "unimport_errors": {},
   "connection_errors": {},
   "total_import_errors": "20",
   "total_unimport_errors": "0",
   "total_connection_errors": "0"
  }
 }
},
"error_details_last_window": [
 {
  "timestamp": {
   "seconds": "1736757948",
   "nanos": 206000000
  },
  "category": "IMPORT_ERROR",
  "import_error_type": "IET_PERMISSION_ERROR",
  "node_id": 1,
  "description": "Import failed: PERMISSION_DENIED",
  "driver_error": "Requested DMA is in use [NV_ERR_DMA_IN_USE]",
  "event_id": "1"
 }
]