dcgmi stats#

NAME#

dcgmi-stats - collect and report GPU process and job statistics

SYNOPSIS#

dcgmi stats --enable [--group <group-id>]
            [--update-interval <milliseconds>]
            [--max-keep-age <seconds>] [--host <address>]
dcgmi stats --disable [--group <group-id>] [--host <address>]
dcgmi stats --pid <pid> [--group <group-id>] [--verbose]
            [--host <address>]
dcgmi stats --jstart <job-id> [--group <group-id>] [--host <address>]
dcgmi stats --jstop <job-id> [--host <address>]
dcgmi stats --job <job-id> [--verbose] [--host <address>]
dcgmi stats --jremove <job-id> [--host <address>]
dcgmi stats --jremoveall [--host <address>]

DESCRIPTION#

dcgmi stats controls the watches used for process and job accounting and reports data collected from those watches. Process reports are selected by operating-system PID. Job reports use an arbitrary caller-provided job ID and cover the interval between --jstart and --jstop.

Statistics can include execution time, energy, maximum memory use, utilization, clocks, PCIe traffic, ECC and XID events, and time spent in clock-slowdown states. Available fields depend on the GPU and driver.

OPTIONS#

Watch management#

-e, --enable: Enable process watches for the selected group and begin recording data.
-d, --disable: Check that the selected group exists, then print that process-statistics watches cannot be disabled. In DCGM 4.6 this operation does not stop any watches and still returns success when the group lookup succeeds.
-u <milliseconds>, --update-interval <milliseconds>: With --enable, set the sampling interval. The default is 30,000 ms.
-m <seconds>, --max-keep-age <seconds>: With --enable, retain samples for this many seconds. The default is 3,600 seconds. Choose a value longer than the processes or jobs to report.

Process reporting#

-p <pid>, --pid <pid>: Display statistics for a process. The PID must be nonnegative.

Job accounting#

-s <job-id>, --jstart <job-id>: Mark the beginning of a job on the selected group. The caller-chosen ID can contain at most 64 characters and must not already be in use.
-x <job-id>, --jstop <job-id>: Mark the end of a previously started job. Job IDs are at most 64 characters.
-j <job-id>, --job <job-id>: Display retained statistics for a job. Job IDs are at most 64 characters.
-r <job-id>, --jremove <job-id>: Remove retained statistics for one job. Job IDs are at most 64 characters.
-a, --jremoveall: Remove retained statistics for every job.

Targets and output#

-g <group-id>, --group <group-id>: Select a persistent GPU group for watch management, process reporting, or --jstart. The default is the built-in all-GPU group. A job ID identifies the group implicitly for later job operations.
-v, --verbose: With --pid or --job, include per-GPU process details.
--host <address>: Connect to the specified host engine.
-h, --help: Print generated usage information and exit.

COLLECTION LIFECYCLE#

Enable watches before the workload starts. --enable requests the process-accounting field set for the group and keeps that request active in the host engine after dcgmi exits. The watch configuration and recorded jobs are in-memory host-engine state and do not survive a host-engine restart. DCGM 4.6 does not implement disabling these watches: --disable only prints Process stats watches cannot be disabled at this time. Once enabled, the request therefore remains until the relevant host-engine state is torn down.

For scheduler accounting, call --jstart after watches have been enabled and immediately before launching the workload, then call --jstop after the workload ends. A job ID maps to its group at start time, which is why later stop, report, and remove operations do not take --group. Retain samples longer than the complete interval to be reported. --jremove and --jremoveall delete retained job records explicitly.

REPORTED STATISTICS#

The exact values available depend on the watched fields and GPU. N/A means the host engine did not retain a supported value for that statistic.

Process and job report contents#
Section	`--pid`	`--job`
Execution	Start, end, duration, number of conflicting processes; verbose output includes conflicting compute and graphics PIDs	Start, end, duration, and number of compute plus graphics processes
Performance	Energy consumed in joules and maximum GPU memory in bytes	Energy consumed, average/maximum/minimum power in watts, and maximum GPU memory in bytes
Per-GPU performance with `--verbose`	Average/maximum/minimum SM clock, memory clock, SM utilization, memory utilization, and PCIe receive/transmit bandwidth in megabytes	The same per-GPU clock, utilization, and PCIe summaries
Events	Double-bit ECC errors, PCIe replay warnings, and critical XID count	Single- and double-bit ECC errors, PCIe replay warnings, and critical XID count
Event detail with `--verbose`	Critical-XID times relative to process start	Critical-XID times relative to job start
Clock slowdown	Percentage of runtime attributed to power, thermal, reliability, board limit, low utilization, and sync boost	The same six percentages
Process utilization with `--verbose`	PID and average SM and memory utilization for each GPU	Compute and graphics PIDs with average SM and memory utilization for each GPU
Health	Overall health; verbose output includes each incident system	Overall health; verbose output includes each incident system

Summary output aggregates across the selected GPUs. Use --verbose when a per-GPU value or the identity of a conflicting process or incident is needed. For the programmatic structures and collection model, see Process Statistics and Job Statistics.

EXAMPLES#

Enable watches for group 1 with five-second sampling and two-hour retention:

$ dcgmi stats --enable --group 1 \
    --update-interval 5000 --max-keep-age 7200

Display statistics for PID 1234:

$ dcgmi stats --pid 1234 --group 1 --verbose

Record and report a scheduler job:

$ dcgmi stats --jstart batch-4815 --group 1

$ dcgmi stats --jstop batch-4815

$ dcgmi stats --job batch-4815 --verbose