Process and Job Statistics#

DCGM can summarize GPU activity for an operating-system process or for a caller-defined scheduler job. The service is built on core field watches: the host engine must collect and retain the required GPU data before it can calculate a result for a completed interval.

This feature is convenient for node-level inspection and scheduler epilogues. It is not a durable accounting database. Export the result if it must survive a host-engine restart or be joined across nodes.

For exact controls and report contents, see dcgmi stats.

Choose the Time Boundary#

Process and job reports select different windows.

Statistics modes#
Mode	Identity	Time boundary	Use it when
Process	One operating-system PID.	The process activity observed in retained samples.	The PID is known and represents the workload of interest.
Job	An arbitrary caller-provided job ID associated with a GPU group.	Explicit `--jstart` and `--jstop` calls.	A scheduler must aggregate a parent process, children, or several PIDs over one allocation window.

A process report can include execution time, energy, maximum GPU memory, clocks, utilization, PCIe traffic, ECC or XID events, slowdown reasons, conflicting processes, and health information. A job report aggregates corresponding data across the PIDs and GPUs observed during the marked window. Hardware and field support determine which values are available.

Neither mode discovers which scheduler allocation a PID belongs to. The integration supplies the GPU group and, for job mode, the job boundary.

Enable Collection Before Work#

Create or identify the GPU group first. Then enable statistics collection before launching the workload:

$ dcgmi stats --enable --group <group-id> \
    --update-interval 5000 --max-keep-age 7200

--update-interval is in milliseconds. --max-keep-age is in seconds. The example samples every five seconds and retains up to two hours of data.

Choose these values deliberately:

The interval must be short enough for the shortest activity that matters. A process that starts and ends between samples might not contribute the expected detail.
Retention must exceed the workload duration plus prologue, epilogue, and reporting delay.
Shorter intervals and longer retention consume more host-engine work and memory.
Collection begins after the watch is enabled; DCGM cannot reconstruct a period that was never sampled.

The default command-line interval is 30 seconds and the default retention is 3,600 seconds. Those defaults are not automatically appropriate for every batch queue.

Privilege and accounting prerequisites#

Process accounting can require root privileges to enable driver accounting mode. Run the host engine with the credentials required by the platform. If DCGM reports DCGM_ST_REQUIRES_ROOT, an administrator can enable GPU accounting mode with nvidia-smi -am 1 before starting DCGM, then repeat the watch setup. Elevating only the dcgmi client does not change the credentials of an already-running host engine.

Watch lifetime in DCGM 4.6#

dcgmi stats --enable leaves the process-statistics watches active in the host engine after the command exits. DCGM 4.6 does not implement disabling them: dcgmi stats --disable reports that process-statistics watches cannot be disabled and does not stop collection.

Plan this as node-agent state rather than a temporary interactive watch. Statistics watches and retained jobs are cleared when the relevant host-engine state is torn down.

Report One Process#

After collection has been enabled, request the retained report for a PID:

$ dcgmi stats --pid 1234 --group <group-id> --verbose

The group limits the GPUs examined for that PID. --verbose adds per-GPU details, process identities, event timing, and health incidents where available.

Use process mode when the PID is the correct accounting unit. If a launcher creates many children, processes migrate among the selected GPUs, or the scheduler needs one explicit allocation window, use job mode instead.

Record a Scheduler Job#

The scheduler should own the complete lifecycle.

Prologue#

Discover the assigned GPUs and correlate them with UUIDs.
Create a DCGM group that mirrors the allocation.
Enable statistics watches with retention longer than the expected job.
Start the job record immediately before launching the workload:
```
$ dcgmi stats --jstart batch-4815 --group <group-id>
```

The caller chooses the job ID. It can contain at most 64 characters and must not already identify another retained record. At start time, DCGM associates the ID with the selected group.

Epilogue#

Stop the record after the workload and its relevant child processes have ended:

$ dcgmi stats --jstop batch-4815

Read the summary before destroying the group or restarting the host engine:

$ dcgmi stats --job batch-4815 --verbose

Later job operations do not take a group ID because the association was made by --jstart. Stopping a job freezes its interval but retains the record for queries. Remove the record when the result has been exported:

$ dcgmi stats --jremove batch-4815
$ dcgmi group --delete <group-id>

--jremoveall removes every retained job and should be reserved for a controller that intentionally owns the whole host-engine job namespace.

Interpret the Summary#

Reported values combine several kinds of evidence:

process facts, such as start, end, runtime, memory high-water mark, and observed PIDs;
sampled gauges, summarized as minimum, maximum, or average;
counter deltas, such as energy or errors, over the process or job window;
time attributed to clock-slowdown reasons; and
health incidents reported for the selected GPUs.

Keep these limits in mind:

Sampling resolution: A report cannot be more precise than the retained input data. Very short spikes or processes can fall between updates.
Entity support: dcgmi stats is a GPU process and job workflow. A mixed entity group does not turn it into CPU, NVSwitch, or application-level accounting.
Unavailable data: N/A means the host engine did not retain a supported value for that statistic. Check field support, privilege, watch timing, and retention.
Aggregation: A group-level number can hide imbalance among GPUs. Use --verbose when placement, one bad device, or a conflicting process matters.
Failure boundaries: A host-engine restart loses in-memory watches and job records. A scheduler integration should treat restart detection and state recreation as part of its prologue logic.

DCGM summarizes GPU management data; it does not replace the scheduler’s authoritative job database, CPU and memory accounting, container isolation, or application-level performance metrics.