CPU Monitoring#

The system-monitoring module is module ID 9. It discovers supported NVIDIA CPUs and their cores and supplies CPU telemetry through the same field-watch model that DCGM uses for GPUs. The module is loaded when system-monitoring functionality is first requested.

When CPU monitoring is useful#

CPU monitoring is useful on NVIDIA Grace systems where GPU behavior must be understood together with its host-side workload. It can help answer whether a job is CPU-bound, whether selected cores are saturated, whether CPU power or temperature is approaching a platform limit, and whether CPU clock behavior correlates with a GPU performance change.

The module provides telemetry, not a general-purpose host monitor. It covers the NVIDIA CPU entities and fields that DCGM exposes; it does not replace operating-system tools for processes, memory, storage, or networking. A typical node workflow combines DCGM CPU and GPU fields under the same monitoring system, then uses OS-level tools when a CPU signal needs process- level investigation.

Requirements and entity model#

CPU monitoring is supported for NVIDIA Grace CPUs, beginning with DCGM 3.3. On unsupported hosts, CPU discovery is empty and CPU field watches are not applicable.

DCGM uses two entity groups:

  • DCGM_FE_CPU, selected as cpu:<id>, represents a whole CPU.

  • DCGM_FE_CPU_CORE, selected as core:<id>, represents one CPU core.

Temperature and power are CPU-scoped. Utilization and clock fields are core- scoped. Pairing a valid field with the wrong entity type can produce an error or an unavailable value.

Discover CPU and core IDs at runtime:

$ dcgmi discovery --list
$ dcgmi discovery --info a --cpuid 0

The CPU information report includes the core ranges owned by that CPU. Core IDs should not be inferred from the CPU ID or copied from another host.

CPU and core fields#

CPU monitoring fields#

Field name

Description

ID

Entity

DCGM_FI_DEV_CPU_UTIL_TOTAL

Total percentage of time in use

1100

Core

DCGM_FI_DEV_CPU_UTIL_USER

Percentage of time used by user processes

1101

Core

DCGM_FI_DEV_CPU_UTIL_NICE

Percentage of time used by low-priority processes

1102

Core

DCGM_FI_DEV_CPU_UTIL_SYS

Percentage of time used by the system

1103

Core

DCGM_FI_DEV_CPU_UTIL_IRQ

Percentage of time used for interrupts

1104

Core

DCGM_FI_DEV_CPU_TEMP_CURRENT

Current temperature in degrees Celsius

1110

CPU

DCGM_FI_DEV_CPU_CLOCK_CURRENT

Current clock speed in kHz

1120

Core

DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT

Current power use in watts

1130

CPU

DCGM_FI_DEV_CPU_POWER_LIMIT

Current power limit in watts

1131

CPU

DCGM_FI_DEV_CPU_VENDOR

CPU vendor name

1140

CPU

DCGM_FI_DEV_CPU_MODEL

CPU model name

1141

CPU

Example: correlate CPU pressure with a workload#

Suppose discovery reports CPU 0 and cores 0 through 3 are the cores assigned to a workload. In one terminal, monitor CPU temperature, power, and power limit for ten samples:

$ dcgmi dmon --entity-id cpu:0 \
    --field-id 1110,1130,1131 --count 10

In another terminal, monitor the assigned cores at the same one-second cadence:

$ dcgmi dmon --entity-id 'core:{0-3}' \
    --field-id 1100,1101,1103,1120 --delay 1000 --count 10

Run the application during the sampling window. Sustained high total utilization on the assigned cores together with lower-than-expected GPU activity can justify investigating a CPU-side bottleneck. Temperature near a platform limit, power near the reported power limit, and a falling core clock can provide additional context, but none of those signals alone proves the cause of application performance.

For a reusable selection, create a mixed group from the IDs reported by discovery:

$ dcgmi group --create job-4815 \
    --add 'cpu:0,core:{0-3},gpu:0'

Not every consuming command accepts every entity type or every mixture. A group is a reusable selector, not a guarantee that one field applies to all of its members. Direct entity selections are often clearer when CPU- and core-scoped fields differ.

Further reading#