dcgmi#

NAME#

dcgmi - manage and monitor NVIDIA GPUs and other DCGM entities

SYNOPSIS#

dcgmi <command> [<options>]
dcgmi -v [--host <address>]
dcgmi --version
dcgmi --help

DESCRIPTION#

dcgmi sends management and monitoring requests to an NVIDIA DCGM host engine. Unless a command says otherwise, it connects to the host engine on localhost and operates on the default all-GPU group.

COMMANDS#

Discovery and organization#

discovery: List DCGM entities and query device attributes. See dcgmi discovery.
group: Create and manage groups of DCGM entities. See dcgmi group.
fieldgroup: Create and manage reusable groups of DCGM field IDs. See dcgmi fieldgroup.

Management and health#

config: Get, set, and enforce GPU configuration. See dcgmi config.
policy: Configure policy actions and listen for policy violations. See dcgmi policy.
health: Configure background health watches and check their results. See dcgmi health.
set: Change host-engine settings or attach and detach the driver. See dcgmi set.

Diagnostics#

diag: Run local active health diagnostics. See dcgmi diag.
mndiag: Run a coordinated diagnostic across multiple hosts. See dcgmi mndiag.

Monitoring and inspection#

stats: Manage process and job watches and report their statistics. See dcgmi stats.
dmon: Stream field values for selected entities. See dcgmi dmon.
topo: Display GPU topology and group topology summaries. See dcgmi topo.
nvlink: Display NVLink status and GPU NVLink error counters. See dcgmi nvlink.
profile: List supported profiling metric groups or control profiling collection. See dcgmi profile.
introspect: Display resource use by the host engine. See dcgmi introspect.
modules: List or denylist host-engine modules. See dcgmi modules.

Testing and validation#

test: Inspect cache metadata, inject synthetic field values, or pause and resume metric collection in an isolated test environment. See dcgmi test.

COMMON CONVENTIONS#

Host addresses#

Commands that accept --host use localhost by default. An address can be an IP address, a fully qualified domain name, a host and port, or a Unix socket or VSOCK URI:

localhost
host.example.com
host.example.com:5555
unix:///run/nvidia-dcgm/nv-hostengine.sock
vsock://3:5555

Use an absolute Unix socket path so client and host-engine working directories cannot change its meaning. The host engine must be listening on the selected transport. A VSOCK address uses vsock://<cid>:<port>; the default port is 5555 when it is omitted.

Note

A TCP address can cross a host boundary and is subject to routing and firewall policy; a Unix-socket path is host-local and relies on directory and socket permissions. See the host-engine listener security note for the DCGM exposure model and links to the underlying Linux socket documentation.

Entity selectors#

Commands with --entity-id accept a comma-separated entity list. A bare number denotes a GPU. An explicit selector has the form <entity-type>:<entity-id>.

Entity selector names#
Entity type	Short name	Example
`gpu`	`g`	`gpu:0` or `0`
`instance`	`i`	`instance:1`
`compute_instance`	`c`	`compute_instance:2`
`vgpu`	`v`	`vgpu:4`
`nvswitch`	`n`	`nvswitch:0`
`cpu`	`--`	`cpu:0`
`core`	`--`	`core:12`
`cx`	`--`	`cx:0`
`link`	`l`	`link:259` or `link:0x103`
`gpu_link`	`--`	`gpu_link:0:5`
`switch_link`	`--`	`switch_link:0:5`

Braces select inclusive numeric ranges, for example gpu:{0-3}, cpu:{0-1}, or gpu_link:{0-1}:{0-5}. The following wildcard forms are also accepted where the command supports the corresponding entity type:

*                 All GPUs
cpu:*             All CPUs
*/*               All MIG GPU instances
*/*/*             All MIG compute instances
0/*               All GPU instances on GPU 0
0/*/0             Compute instance 0 in every GPU instance on GPU 0

Not every command can operate on every entity type. The subcommand page states any narrower requirement. Use ENTITY ID DISCOVERY to obtain the IDs currently assigned by the selected host engine.

Groups#

Persistent groups have numeric IDs and are managed with dcgmi group. Discover the current IDs with dcgmi group --list rather than hard-coding them. Several commands default to the built-in all-GPU group. Commands that accept named built-in groups document their own names because the accepted spellings are not uniform across all subcommands.

JSON output#

-j or --json changes output to JSON only for the command forms that list it in their synopsis. Other command forms either produce no structured result or support text output only.

Warning

DCGM does not publish a stable, versioned schema for dcgmi JSON output. Treat field names, nesting, types, and presence as release-specific: pin and test the producing DCGM release, validate required fields, and tolerate additional fields where practical. Use the C/C++ API when automation needs a durable compatibility contract.

GLOBAL OPTIONS#

--version: Print the version of the local dcgmi binary and exit. This option does not connect to a host engine.
-v, --vv: Display detailed local and host-engine build information. In DCGM 4.6.0, this form only produces the report after it connects to the host engine.
--host <address>: Connect to the specified host engine. With -v, this selects the host engine whose build information is shown. For subcommands, see the command’s synopsis.
-h, --help: Print generated usage information and exit.
--, --ignore_rest: Stop parsing labeled options. This is a parser compatibility feature and is not needed in normal dcgmi use.

EXIT STATUS#

0 indicates that the requested operation completed successfully. A nonzero status indicates a parser, connection, or DCGM operation failure.

dcgmi returns DCGM’s signed dcgmReturn_t value directly from main(). POSIX shells expose only the low eight bits of that value, so a DCGM status of -1 appears in $? as 255 and a command-line parse failure of -2 appears as 254. Use the status name and error text that dcgmi prints as the authoritative failure reason; do not interpret the shell value as a signed DCGM status.

The complete signed dcgmReturn_t catalogue is in Enums and Macros. To determine the shell status for a listed negative value, take it modulo 256. Subcommand pages call out statuses whose meaning is specific to that operation.

If dcgmi is terminated by a signal, POSIX-style shells conventionally report 128 plus the signal number, such as 130 for SIGINT and 143 for SIGTERM. Those values describe process termination rather than a dcgmReturn_t result.

Note

Shell automation sees an unsigned status from 0 through 255, not the full signed value returned by C main(). See Exit Status in the GNU Bash manual and signal(7) for the shell and signal conventions used above.

An operation can succeed while reporting an unavailable value for an individual field or entity. Subcommand pages describe these cases where they affect automation.

ENVIRONMENT#

CUDA_VISIBLE_DEVICES: Setting this variable only in the dcgmi process does not select the GPUs used by DCGM. Set it for nv-hostengine before the host engine starts, or use the entity and group selectors of the requested command. When it is set for dcgmi, the client reports how its value compares with the host engine’s value.

Environment changes are process-local. See the process-inheritance note when deciding whether a value belongs on the client or the systemd-managed host engine.

EXAMPLES#

Print the local binary version:

$ dcgmi --version

List entities discovered by a host engine on another host:

$ dcgmi discovery --host node01.example.com --list

Connect through a Unix socket and list DCGM modules:

$ dcgmi modules --host unix:///run/nvidia-dcgm/nv-hostengine.sock --list